Skip to content

[train] feat: Add OpenClaw online RL/OPD/Combine training path #76

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jamindy wants to merge 2 commits into
base: main
Choose a base branch
from
Open

[train] feat: Add OpenClaw online RL/OPD/Combine training path #76

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
071d50b
[train] feat: Add OpenClaw online RL/OPD/Combine training
jamindy Jul 3, 2026
9581e73
update Sampling params
jamindy Jul 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5,278 changes: 5,278 additions & 0 deletions examples/openclaw/eval/GSM8K.json
View file
Open in desktop

Large diffs are not rendered by default.

244 changes: 244 additions & 0 deletions examples/openclaw/eval/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,244 @@
# OpenClaw-Test: End-to-End Evaluation for OpenClaw-RL Training Methods

This directory contains an automated evaluation suite that tests the **real-world effectiveness** of models trained with OpenClaw-RL method.

## What Does This Test Do?

The evaluation simulates a realistic multi-turn agentic workflow using **GSM8K math problems** as the task domain. An external LLM (the "user") interacts with the OpenClaw agent (your trained model) through the OpenClaw gateway API, testing whether the agent can:

- Read files from its workspace
- Solve math problems with complete step-by-step reasoning
- Follow stylistic instructions (e.g., rewrite in a more natural tone)
- Write results back to files
- Grade existing solutions against ground truth
- Produce detailed, friendly feedback

The test consists of **three sequential phases** (you can also run them together, but you need to obtain homework1 and homework2 first if you want to try this joint optimization):

### Phase 1: Student Chat (`student_chat.py`)

An external LLM role-plays as a **lazy student** who asks the OpenClaw agent to do their homework. For each GSM8K problem:

1. The problem is written to `homework/i.txt` in the OpenClaw workspace.
2. The "student" asks the agent to read the file and solve it.
3. If the agent's answer looks too AI-like (bold text, numbered lists, etc.), the student tells it to rewrite in a more natural style.
4. Once satisfied, the student asks the agent to append the answer to the homework file.
5. The student says `HOMEWORK_DONE` to end the session.

This phase tests the agent's **instruction following**, **math reasoning**, **file I/O**, and **style adaptation** abilities.

### Phase 2: TA Chat (`TA_chat.py`)

An external LLM role-plays as a **TA** who grades the student's submissions. For each problem:

1. If needed, `homework/` is copied to `homework1/` in the OpenClaw workspace.
2. The TA provides the original question and ground truth answer to the agent.
3. The agent reads the student's submission from `homework1/i.txt`, compares it with the correct answer, and writes grading comments.
4. If the comments are too brief or not specific enough, the TA asks for a rewrite.
5. Once satisfied, the TA asks the agent to append the comments to the file.
6. The TA says `GRADING_DONE` to end the session.

This phase tests the agent's **reading comprehension**, **evaluation accuracy**, **feedback specificity**, and **multi-step file operations**.

### Phase 3: Teacher Chat (`teacher_chat.py`)

An external LLM role-plays as a **teacher** who reviews the already graded homework and writes comments about the student's strengths and weaknesses. For each problem:

1. If needed, `homework1/` is copied to `homework2/` in the OpenClaw workspace.
2. The teacher provides the original question and ground truth answer to the agent.
3. The agent reads the graded submission from `homework2/i.txt` and writes friendly, patient feedback about strengths and weaknesses.
4. If the comments are not friendly or patient enough, the teacher asks for a rewrite.
5. Once satisfied, the teacher asks the agent to append the comments to the file.
6. The teacher says `COMMENT_DONE` to end the session.

This phase tests the agent's **review quality**, **tone control**, **supportive feedback**, and **multi-step file operations**.

> **Run order matters:** Run `student_chat.py` first so the homework files contain student solutions, then run `TA_chat.py` to grade them, then run `teacher_chat.py` to add teacher comments.

---

## Architecture Overview

```
┌─────────────────────┐ ┌───────────────────────────────┐
│ External LLM │ │ OpenClaw RL Server │
│ (Student/TA/Teacher) │ │ (your trained model) │
│ Port 30001 │ │ Port 30000 │
│ via launch_user_ │ │ via openclaw-rl/opd/combine │
│ llm.sh or closed- │ │ shell scripts │
│ source API │ │ │
└────────┬────────────┘ └──────────┬────────────────────┘
│ │
│ student/TA/teacher messages │ agent responses
│ │
└──────────┐ ┌────────────────┘
▼ ▼
┌──────────────────┐
│ student_chat.py │
│ TA_chat.py │
│ teacher_chat.py │
│ (orchestrator) │
└──────────────────┘
```

---

## Step-by-Step Guide

### Prerequisites

- A running OpenClaw environment (see the [main README](../README.md))
- Python 3.12 with `requests` and `openai` packages installed
- A `GSM8K.json` dataset file (JSON array with `question` and `ground_truth_answer` fields per entry)
- 4 available GPUs for `run_openclaw_rl_eval_4gpu.sh`: GPU 0,1,2 are used by the RL service, and GPU 3 is used by the external role LLM.

### Step 1: Configure the 4-GPU Evaluation Script

Edit the configurable settings at the top of `run_openclaw_rl_eval_4gpu.sh`:

```bash
ROOT="/path/frameworks/uni-agent"
MODEL_PATH="/path/models/Qwen3-VL-4B-Instruct"
DATASET="${ROOT}/examples/openclaw/eval/GSM8K.json"
```

The script uses the same model path for both the RL service and the user LLM by default:

- RL service endpoint: `http://127.0.0.1:30000`
- user LLM endpoint: `http://127.0.0.1:30001/v1`
- RL API key: `rl-local-token`
- user LLM API key: `user-llm-local-token`
- user LLM model name: `qwen3-vl-4b-user-llm`

You can override the evaluation scale without editing the script:

```bash
export NUM_PROBLEMS=36
export MAX_TURNS=8
export TOTAL_ROLLOUT_STEPS=2000
```

For long conversations, the script also bounds the prompt/reply text sent through the driver:

```bash
export OPENCLAW_DRIVER_HISTORY_MAX_CHARS=20000
export OPENCLAW_AGENT_REPLY_MAX_CHARS=12000
```

### Step 2: Run the Full Pipeline

Run the script from the repository root:

```bash
cd /path/frameworks/uni-agent
bash examples/openclaw/eval/run_openclaw_rl_eval_4gpu.sh
```

The script performs the complete evaluation flow:

1. Cleans up stale `uni_agent.openclaw.rl.train_entry`, `vllm serve`, and Ray processes.
2. Starts the RL service on GPU 0,1,2 by running `examples/openclaw/train_rl.sh`.
3. Waits for `http://127.0.0.1:30000/healthz` to become ready.
4. Starts the user LLM on GPU 3 by running `examples/openclaw/eval/launch_user_llm.sh`.
5. Waits for `http://127.0.0.1:30001/v1/models` to become ready.
6. Exports the OpenClaw gateway and OpenAI-compatible environment variables used by the eval scripts.
7. Runs `student_chat.py`, then `TA_chat.py`, then `teacher_chat.py`.

### Step 3: Check Logs and Outputs

Runtime logs are written under the repository-level `logs/` directory:

- RL service log: `logs/rl_eval_rl_server.log`
- user LLM log: `logs/rl_eval_user_llm.log`

Evaluation outputs are written in this directory:

- Student stage: `results_student.txt`
- TA stage: `results_TA.txt`
- Teacher stage: `results_teacher.txt`

The workspace defaults to `~/.openclaw/workspace`:

- `homework/`: student solutions
- `homework1/`: TA-graded submissions
- `homework2/`: teacher-commented submissions

### Equivalent Manual Eval Commands

After the RL service and user LLM are ready, `run_openclaw_rl_eval_4gpu.sh` runs the three stages with these environment variables:

```bash
export OPENCLAW_GATEWAY_URL="http://127.0.0.1:30000"
export OPENCLAW_GATEWAY_TOKEN="rl-local-token"
export OPENCLAW_ENDPOINT_MODE="gateway"
export OPENCLAW_AGENT_MODEL="default"
export OPENCLAW_WORKSPACE="${HOME}/.openclaw/workspace"

export OPENAI_BASE_URL="http://127.0.0.1:30001/v1"
export OPENAI_API_KEY="user-llm-local-token"
export EXTERNAL_MODEL="qwen3-vl-4b-user-llm"
export OPENCLAW_DRIVER_HISTORY_MAX_CHARS=20000
export OPENCLAW_AGENT_REPLY_MAX_CHARS=12000

python student_chat.py --dataset "${DATASET}" --num-problems "${NUM_PROBLEMS}" --max-turns "${MAX_TURNS}"
python TA_chat.py --dataset "${DATASET}" --num-problems "${NUM_PROBLEMS}" --max-turns "${MAX_TURNS}"
python teacher_chat.py \
--dataset "${DATASET}" \
--num-problems "${NUM_PROBLEMS}" \
--max-turns "${MAX_TURNS}"
```

---

## Command-Line Arguments

`student_chat.py`, `TA_chat.py`, and `teacher_chat.py` accept the same arguments:

| Argument | Default | Description |
|---|---|---|
| `--dataset` | *(required)* | Path to the GSM8K JSON file |
| `--num-problems` | `5` | Number of problems to process |
| `--max-turns` | `8` | Maximum conversation turns per problem |
| `--max-retries` | `3` | Maximum retries per network call |
| `--output` | See below | Output file for the first OpenClaw reply from each problem |

Default output files:

| Script | Default output |
|---|---|
| `student_chat.py` | `results_student.txt` |
| `TA_chat.py` | `results_TA.txt` |
| `teacher_chat.py` | `results_teacher.txt` |

## Environment Variables

| Variable | Required | Default | Description |
|---|---|---|---|
| `OPENCLAW_GATEWAY_TOKEN` | Yes | — | Auth token for the OpenClaw gateway |
| `OPENAI_API_KEY` | Yes | — | API key for the external LLM (student/TA/teacher) |
| `OPENCLAW_GATEWAY_URL` | No | `http://localhost:18789` | OpenClaw gateway base URL. `run_openclaw_rl_eval_4gpu.sh` sets this to `http://127.0.0.1:30000`. |
| `OPENCLAW_ENDPOINT_MODE` | No | `gateway` | OpenClaw endpoint mode used by the eval scripts |
| `OPENCLAW_AGENT_MODEL` | No | `default` | Model name sent to the OpenClaw gateway |
| `OPENCLAW_WORKSPACE` | No | `~/.openclaw/workspace` | Path to the OpenClaw workspace directory |
| `OPENAI_BASE_URL` | No | *(OpenAI default)* | Base URL for the external LLM API. `run_openclaw_rl_eval_4gpu.sh` sets this to `http://127.0.0.1:30001/v1`. |
| `EXTERNAL_MODEL` | No | `gpt-4o` | Model name for the external LLM. `run_openclaw_rl_eval_4gpu.sh` sets this to `qwen3-vl-4b-user-llm`. |
| `OPENCLAW_DRIVER_HISTORY_MAX_CHARS` | No | `20000` in `run_openclaw_rl_eval_4gpu.sh` | Maximum conversation-history characters sent to the external role LLM |
| `OPENCLAW_AGENT_REPLY_MAX_CHARS` | No | `12000` in `run_openclaw_rl_eval_4gpu.sh` | Maximum agent-reply characters forwarded to the external role LLM |

---



## File Structure

```
eval/
├── README.md # This file
├── run_openclaw_rl_eval_4gpu.sh
│ # 4-GPU end-to-end RL eval pipeline
├── launch_user_llm.sh # Script to host the external LLM via vLLM
├── student_chat.py # Phase 1: Student asks agent to solve homework
├── TA_chat.py # Phase 2: TA asks agent to grade homework
├── teacher_chat.py # Phase 3: Teacher asks agent to comment on strengths and weaknesses
└── GSM8K.json # Dataset (to be placed here)
```
151 changes: 151 additions & 0 deletions examples/openclaw/eval/TA_chat.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
"""AI-to-agent GSM8K homework grading runner (TA role).

The external LLM plays a TA who uses the agent under test to grade student
homework. The homework files (homework1/i.txt) are assumed to already exist in
the workspace (copied from a prior student_chat.py run). The TA provides the
original question and ground truth answer, asks the agent to grade, reviews the
feedback, and then has it appended to the file.

The agent under test is reached through ``eval_common.AgentEndpoint`` (gateway or
OpenAI-compatible) -- see ``eval_common`` for environment configuration.

Usage:
python TA_chat.py --dataset GSM8K.json --num-problems 5
"""

import argparse
import os

from eval_common import (
AgentEndpoint,
ensure_homework_dir,
get_external_model,
get_workspace,
load_dataset,
make_external_client,
run_role_session,
)

HOMEWORK_DIR = "homework1"
SOURCE_HOMEWORK_DIR = "homework"

TA_SYSTEM_PROMPT = """\
You are role-playing as a TA who is grading student homework. You talk casually. \
You want the AI to write comments that are detailed and specific.

You CANNOT grade, rewrite, rephrase, or produce any comments yourself. \
You can ONLY tell the AI what to do. Never do the grading yourself. \
You know NOTHING about the homework content — you have not read it and \
you do not know any numbers, steps, or answers from the problem. \
Only the AI can read the file and see what the student wrote. \
You must NEVER mention any specific numbers, calculations, formulas, \
problem details, or student answers in your messages. \
You must NEVER list steps the student should have taken. \
You must NEVER write example comments or sample feedback. \
You must NEVER do any part of the grading yourself. \
Your ONLY job is to give short, general instructions to the AI — \
things like "make it more detailed", "mention whether the student \
showed their work", "check if the student handled all parts of the \
problem". Keep your instructions general and brief.

Your goal: get the AI to grade the student's homework and write comments. \
The comments must be detailed and specific. If the AI's comments are too \
short or not specific enough, tell it to rewrite using short, general phrases. \
Examples of good things to say: \
"Write more specific grading comments." \
"Make your grading more detailed — go deeper into the student's reasoning." \
"Your comments are too short, please expand them." \
"Be more specific about what the student got right and wrong." \
"Add more detail about the steps the student took." \
Just tell it to fix it — don't fix it yourself. \
If the comments are already detailed and specific, no need to rewrite.

Steps:
1. Look at what the AI gives you. If the comments are not detailed or specific enough, \
tell it to redo it in short, general terms (e.g. "make your grading more specific" \
or "add more detail about the student's reasoning" — NOT "the student should have \
done X then Y"). \
Do NOT mention writing to the file in the same message. Only ask for a rewrite.
2. After the AI shows you the satisfactory version, THEN in a \
separate message ask it to append the comments to the end of the homework file \
(not overwrite it). Do NOT combine a rewrite request and a write request.
3. After the AI says it saved the file, say exactly: GRADING_DONE

Never say GRADING_DONE until the AI confirms it wrote the file.
Never write or grade anything yourself. Never reference specific problem content. \
Just give short, general instructions."""

FIRST_MESSAGE_TEMPLATE = """\
I'm grading a student's homework. The submission is in the file homework1/{index}.txt \
in your workspace. Please read the file first.

Here is the original question and the correct answer for reference:

Question: {question}

Correct answer: {ground_truth}

Please read the student's submission from the file, compare it with the correct \
answer, and write the grading comments directly. No intro, no summary, no \
"here are the comments" — just the comments themselves, as if you are writing \
them on the student's paper. \
Show me the comments first — don't write to the file until I tell you to."""

DONE_SENTINEL = "GRADING_DONE"


def main():
parser = argparse.ArgumentParser(description="GSM8K homework grading runner via the agent under test (TA role)")
parser.add_argument("--dataset", type=str, required=True, help="Path to GSM8K.json")
parser.add_argument("--num-problems", type=int, default=5, help="Number of problems to grade (default: 5)")
parser.add_argument("--max-turns", type=int, default=8, help="Max turns per problem (default: 8)")
parser.add_argument("--max-retries", type=int, default=3, help="Max retries per network call (default: 3)")
parser.add_argument("--output", type=str, default="results_TA.txt", help="Output file for agent replies")
args = parser.parse_args()

endpoint = AgentEndpoint.from_env()
external_client = make_external_client()
model = get_external_model()
workspace = get_workspace()

ensure_homework_dir(workspace, SOURCE_HOMEWORK_DIR, HOMEWORK_DIR)

problems = load_dataset(args.dataset)
count = min(args.num_problems, len(problems))
print(f"Loaded {len(problems)} problems from {args.dataset}")
print(f"Grading {count} problems, max {args.max_turns} turns each | endpoint mode: {endpoint.mode}\n")

open(args.output, "w").close()
print(f"Output file: {args.output} (cleared)\n")

results = []
for i in range(count):
question = problems[i].get("question", "")
ground_truth = problems[i].get("ground_truth_answer", "?")
completed = run_role_session(
session_id=f"ta-grade-{i}-{os.getpid()}",
endpoint=endpoint,
external_client=external_client,
model=model,
system_prompt=TA_SYSTEM_PROMPT.replace("homework file", f"file {HOMEWORK_DIR}/{i}.txt"),
first_message=FIRST_MESSAGE_TEMPLATE.format(index=i, question=question, ground_truth=ground_truth),
done_sentinel=DONE_SENTINEL,
role_label=f"Grading problem {i} (ground truth: {ground_truth})",
max_turns=args.max_turns,
max_retries=args.max_retries,
output_file=args.output,
)
results.append(completed)

done = sum(results)
print(f"\n{'#' * 60}")
print(f"# Summary: {done}/{count} problems graded within turn limit")
print(f"{'#' * 60}")
for i, ok in enumerate(results):
status = "done" if ok else "incomplete"
gt = problems[i].get("ground_truth_answer", "?")
print(f" Problem {i}: {status} (ground truth: {gt})")


if __name__ == "__main__":
main()
Loading