Skip to content

RUNBOOK.md

Latest commit

 

History

History
148 lines (124 loc) · 6.75 KB

RUNBOOK.md

File metadata and controls

148 lines (124 loc) · 6.75 KB
Raw

RUNBOOK — train qwen3.6-27b on τ²-bench banking (one step at a time)

Run these in order. After each, check the "✅ expect" line before moving on. You run every command yourself (Claude does not). Steps 1–5 are local & free; steps 6+ call Fireworks (cost money). Stop at the first thing that doesn't match.

Legend: 💻 local/free · ☁️ hits Fireworks (paid) · 📄 file you edit

Phase 0 — get the pieces in place (local, free)

Step 1 — install the SDK 💻

pip install eval-protocol firectl
eval-protocol --version && firectl version

✅ expect: both print a version. (Right now eval-protocol is NOT installed.)

Step 2 — find your real qwen3.6-27b base-model path 💻

qwen3.6-27b is YOUR custom model, not a public one. Get its resource name:

export FIREWORKS_API_KEY="fw_..."          # from app.fireworks.ai
firectl get deployment b2f04w5c            # b2f04w5c = your qwen3.6-27b deployment
# look for the model it serves, then:
firectl list models | grep -i qwen

✅ expect: a path like accounts/<your-acct>/models/qwen3p6-27b. Copy it. Then set it everywhere:

export BASE_MODEL="accounts/<your-acct>/models/qwen3p6-27b"
export POLICY_MODEL="fireworks_ai/$BASE_MODEL"

⚠️ Also confirm it's tunable: Fireworks RFT is easiest on ≤~16B. A 27B may need a quota/eligibility check — ask Fireworks or check the Model Library if --go rejects it.

Step 3 — build the dataset 💻 📄 workspace/build_banking_easy.py

python3 workspace/build_banking_easy.py --buckets Easy Medium

✅ expect: Wrote 25 rows -> FIREWORKS_TRAINING/data/banking_easy_medium.jsonl (Easy alone = only 5 rows, too few. Point the evaluator at whichever file you build: export EP_DATASET=FIREWORKS_TRAINING/data/banking_easy_medium.jsonl)

Step 4 — sanity-check the reward math 💻 📄 workspace/score_from_reward_info.py

python3 workspace/score_from_reward_info.py

✅ expect: 5 printed lines — partial credit < 1.0 for the not-matched case, the looping case scored lower, full solve == 1.0, early-stop on loop? True. This proves the reward + loop-guard logic works before any model is involved.

Phase 1 — make the rollout run locally (local orchestration; model calls hit Fireworks)

Step 5 — write the MCP server 📄 FIREWORKS_TRAINING/banking_mcp/server.py

This is the one piece that must match EP's McpGym API exactly, so copy the installed example and swap the domain instead of guessing:

python -c "import eval_protocol, os; print(os.path.dirname(eval_protocol.__file__))"
# find examples/tau2_mcp/server.py under the SDK (or in the python-sdk repo) and copy it:
mkdir -p FIREWORKS_TRAINING/banking_mcp
cp <sdk>/examples/tau2_mcp/server.py FIREWORKS_TRAINING/banking_mcp/server.py

Then edit it to mount OUR env (entrypoint is tau2.domains.banking_knowledge.environment.get_environment, registered domain name "banking_knowledge") instead of airline. See OVERVIEW §3. ✅ expect: python FIREWORKS_TRAINING/banking_mcp/server.py starts and serves the banking tools.

Step 6 — local eval, no training ☁️ (cheap: a few inferences) 📄 FIREWORKS_TRAINING/test_banking.py

ep local-test          # or: bash FIREWORKS_TRAINING/training_run.sh --localtest

✅ expect: a handful of rollouts run end-to-end and print per-row scores. 🔧 First run will likely fail at the two TODO[EP-schema] spots in score_from_reward_info.py (_extract_reward_info / _extract_tool_calls). Print one real row, see where reward_info and the tool calls actually live, fix those two functions, re-run until scores look sane. This is the main glue step.

Step 7 — baseline number ☁️

With Step 6 working, record qwen3.6-27b's mean score + pass-rate on the dataset. This is the bar RFT must beat. Cross-check against your existing leaderboard.

Phase 2 — train on Fireworks ☁️ (paid)

Step 8 — dry-run the launcher (no cost) 💻 📄 FIREWORKS_TRAINING/training_run.sh

bash FIREWORKS_TRAINING/training_run.sh

✅ expect: preflight passes, prints the exact eval-protocol create rft command. If it errors on REPLACE_ME, you skipped Step 2.

Step 9 — launch the RFT smoke run ☁️

bash FIREWORKS_TRAINING/training_run.sh --go

What it runs (qwen3.6-27b, Easy+Medium, group size 16, DAPO, 1 epoch):

eval-protocol create rft \
  --base-model  $BASE_MODEL \
  --output-model qwen3p6-27b-banking-rft-v1 \
  --n 16 --rl-loss-method dapo --temperature 0.8 \
  --max-tokens 4096 --epochs 1 --lora-rank 8 \
  --wandb-project tau2-banking-rft

✅ expect: a job id. Track it:

eval-protocol list           # or watch W&B project tau2-banking-rft

Goal of the smoke run: confirm the loop runs and mean reward moves up — not SOTA.

Step 10 — deploy & re-measure ☁️

firectl create deployment accounts/<your-acct>/models/qwen3p6-27b-banking-rft-v1
# point POLICY_MODEL at the new model and re-run Step 6/7 to measure the lift.

Where you are right now

  • ✅ done: reward scorer, dataset builder, evaluator, launcher, this runbook.
  • ⏳ you do next: Step 1 (install) → Step 2 (find base path) → Step 3 (build) → Step 4 (self-test). Those four are local and unblock everything. Step 5 (MCP server) + Step 6 (fix the 2 schema TODOs) are the real integration work before training.

Serverless vs. your deployment (qwen3.6-27b is NOT serverless)

Confirmed via workspace/check_serverless.py: accounts/fireworks/models/qwen3p6-27b returns NOT_FOUND serverless — it's dedicated-only (deployment b2f04w5c). But the deployment is needed in very few places:

Step Needs deployment? Note
local plumbing test (Step 6) ❌ use a serverless stand-in: export POLICY_MODEL=fireworks_ai/accounts/fireworks/models/gpt-oss-20b
RFT training (Step 9) ❌ Fireworks serves the base internally during create rft
standalone baseline of qwen3.6-27b (Step 7) ✅ only place — warm with firectl deployment update b2f04w5c --min-replica-count 1, scale to 0 after
serving the trained LoRA (Step 10) ✅ non-serverless base ⇒ LoRA addon also non-serverless

POLICY_MODEL is an env var read by test_banking.py — flip it freely (serverless stand-in ↔ deployment) without editing files. For a fully deployment-free loop, train a base that is both serverless and tunable (smaller serverless Qwen3/Llama) instead of 27B.

Still open / risks

  • 27B tunable-tier eligibility on Fireworks RFT (Step 2 caveat).
  • The model.request scope / API-key item in ../TODO.md — RFT rollouts use that path.
  • EP McpGym API not verified (Step 5 copies the installed example to avoid guessing).
  • The 2 TODO[EP-schema] extraction points (Step 6).