Skip to content

Updating INSTRUCTIONS.md and adding docs/opencode-agent.md #410

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
DhavalRepo18 merged 1 commit into from
Jun 25, 2026
Merged

Updating INSTRUCTIONS.md and adding docs/opencode-agent.md #410

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions INSTRUCTIONS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ query="What is the current date and time? Also list assets at site MAIN. Also ge

## Agents

Six runners are available as CLIs registered by `uv sync`; five use MCP tools, while `direct-llm-agent` is a model-only baseline that makes a direct LiteLLM call without MCP tools, planning, retrieval, or code execution. Each is a CLI registered by `uv sync` that takes a single positional `question` argument and spawns the MCP servers as stdio subprocesses on demand.
Seven runners are available as CLIs registered by `uv sync`; six use MCP tools, while `direct-llm-agent` is a model-only baseline that makes a direct LiteLLM call without MCP tools, planning, retrieval, or code execution. Each is a CLI registered by `uv sync` that takes a single positional `question` argument and spawns the MCP servers as stdio subprocesses on demand.

| Runner | Source | Loop | Default model |
| -------------- | ---------------------------- | ------------------------------------------------------------- | ----------------------------------------------------------- |
Expand All @@ -163,9 +163,10 @@ Six runners are available as CLIs registered by `uv sync`; five use MCP tools, w
| `openai-agent` | `src/agent/openai_agent/` | [`openai-agents`](https://github.com/openai/openai-agents-python) SDK Runner | `litellm_proxy/azure/gpt-5.4` |
| `deep-agent` | `src/agent/deep_agent/` | [LangChain deep-agents](https://docs.langchain.com/oss/python/deepagents/overview) (LangGraph), MCP bridged via `langchain-mcp-adapters` | `litellm_proxy/aws/claude-opus-4-6` |
| `stirrup-agent` | `src/agent/stirrup_agent/` | [Stirrup](https://github.com/ArtificialAnalysis/Stirrup) agent loop (in-process), MCP via its `MCPToolProvider`; **code-capable** (writes/runs Python) | `watsonx/meta-llama/llama-4-maverick-17b-128e-instruct-fp8` |
| `opencode-agent` | `src/agent/opencode_agent/` | [OpenCode](https://opencode.ai/docs/) CLI agent loop, MCP via generated OpenCode config; web/bash/edit denied by default | `opencode/gpt-5.1-codex` |
| `direct-llm-agent` | `src/agent/direct_llm_agent/` | Single direct LLM call, no MCP tools, planning, retrieval, or code execution | `litellm_proxy/Azure/gpt-5-mini-2025-08-07` |

- [Agents](#agents) — Stirrup specifics in [docs/stirrup-agent.md](docs/stirrup-agent.md)
- [Agents](#agents) — Stirrup specifics in [docs/stirrup-agent.md](docs/stirrup-agent.md); OpenCode specifics in [docs/opencode-agent.md](docs/opencode-agent.md)

### Usage

Expand All @@ -175,6 +176,7 @@ uv run claude-agent "$query"
uv run openai-agent "$query"
uv run deep-agent "$query"
uv run stirrup-agent "$query"
uv run opencode-agent "$query"
uv run direct-llm-agent "$query"
```

Expand All @@ -199,6 +201,9 @@ uv run direct-llm-agent "$query"
| `--code-enabled` / `--no-code` | stirrup-agent | Enable (default) / disable code execution — selects the code track |
| `--code-backend B` | stirrup-agent | Code sandbox: `docker` (default), `local`, or `e2b` |
| `--max-tokens N` | stirrup-agent | Max output tokens per call; keep under provider limit (default 16384) |
| `--max-steps N` | opencode-agent | Max OpenCode agentic iterations (default: 30) |
| `--attach URL` | opencode-agent | Attach to a running `opencode serve` instance |
| `--allow-bash` / `--allow-edit` / `--allow-web` | opencode-agent | Opt into shell, file edits, or web access; all denied by default |

### Examples

Expand All @@ -224,6 +229,9 @@ uv run stirrup-agent --no-code --show-trajectory \
STIRRUP_CODE_IMAGE=assetops-code \
uv run stirrup-agent --code-backend docker "$query"

# OpenCode CLI-backed agent, using TokenRouter
uv run opencode-agent --model-id tokenrouter/MiniMax-M3 --show-trajectory "$query"

# Direct model-only baseline, no MCP tools
uv run direct-llm-agent --model-id litellm_proxy/Azure/gpt-5-mini-2025-08-07 \
'Return only JSON: {"test": 1}'
Expand Down
301 changes: 301 additions & 0 deletions docs/opencode-agent.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,301 @@
# OpenCode Agent

The `opencode-agent` runner drives [OpenCode](https://opencode.ai/docs/) against
the AssetOpsBench MCP servers. It is a peer to `plan-execute`, `claude-agent`,
`openai-agent`, `deep-agent`, and `stirrup-agent`: same CLI contract, same
persisted `Trajectory`, scored by the same `uv run evaluate`.

OpenCode is a general CLI coding agent, but this runner is configured as a
**tools-first benchmark agent** by default. It can call the AssetOpsBench MCP
servers, read/search local files, and return an answer. Shell commands, file
edits, web access, and follow-up questions are denied unless explicitly enabled.

## Contents

- [Why OpenCode](#why-opencode)
- [Install](#install)
- [Quick start](#quick-start)
- [Model routing](#model-routing)
- [Permissions and web access](#permissions-and-web-access)
- [Headless server mode](#headless-server-mode)
- [CLI flags](#cli-flags)
- [Benchmark-suite usage](#benchmark-suite-usage)
- [Validation runs](#validation-runs)
- [Troubleshooting](#troubleshooting)
- [What was added](#what-was-added)

---

## Why OpenCode

- **CLI-backed agent loop**, so it can be evaluated as an external agent without
embedding another SDK directly in AssetOpsBench.
- **MCP support**, so it connects to the same six AssetOpsBench servers as every
other tool-using runner.
- **Headless automation**, via `opencode run --format json`, which makes it
usable in scripts and benchmark runs.
- **Server/API mode**, via `opencode serve`, which can keep OpenCode warm across
repeated runs.
- **Provider flexibility**, including direct OpenCode providers and
AssetOpsBench router prefixes such as `tokenrouter/` and `litellm_proxy/`.

---

## Install

The Python entry point is declared in `pyproject.toml` as the `opencode-agent`
script. The OpenCode executable itself is installed outside the Python package.

Install the project dependencies from the repo root:

```bash
uv sync
```

Confirm the AssetOpsBench entry point resolved (no model call):

```bash
uv run opencode-agent --help
```

Confirm the OpenCode binary is available:

```bash
opencode --version
```

If `opencode` is missing, install it with one of the methods from the OpenCode
docs, for example:

```bash
npm install -g opencode-ai
```

Run the unit tests for the config and trajectory mapping:

```bash
uv run pytest src/agent/opencode_agent/tests/ -q
```

---

## Quick start

Set the provider credentials for the model route you want, then run a small
question:

```bash
export TOKENROUTER_BASE_URL=https://api.tokenrouter.com/v1
export TOKENROUTER_API_KEY=...

uv run opencode-agent --show-trajectory \
--model-id tokenrouter/MiniMax-M3 \
"What sites are available?"
```

In the `--show-trajectory` output, look for domain tool calls such as
`iot_sites`, `iot_registry_assets`, or `wo_list_workorders`. That confirms
OpenCode discovered and called the AssetOpsBench MCP tools.

> **Quiet runs.** `opencode-agent` runs OpenCode as a subprocess. During long
> questions, the terminal may look quiet until OpenCode finishes. The persisted
> trajectory is written only after the run completes.

---

## Model routing

The runner's default model is `opencode/gpt-5.1-codex`.

| `--model-id` prefix | OpenCode provider config | Required env vars |
| ----------------------- | --------------------------------------- | ----------------- |
| `opencode/<model>` | OpenCode built-in provider | OpenCode auth / provider setup |
| `openai/<model>` | OpenCode built-in OpenAI provider | `OPENAI_API_KEY` or OpenCode auth |
| `anthropic/<model>` | OpenCode built-in Anthropic provider | `ANTHROPIC_API_KEY` or OpenCode auth |
| `tokenrouter/<model>` | Custom OpenAI-compatible `tokenrouter` provider with explicit model registration | `TOKENROUTER_BASE_URL`, `TOKENROUTER_API_KEY` |
| `litellm_proxy/<model>` | Custom OpenAI-compatible `litellm-proxy` provider with explicit model registration | `LITELLM_BASE_URL`, `LITELLM_API_KEY` |

For router prefixes, the runner generates inline OpenCode config using
`OPENCODE_CONFIG_CONTENT`. For example:

```text
tokenrouter/MiniMax-M3 -> tokenrouter/MiniMax-M3
litellm_proxy/azure/gpt-5.4 -> litellm-proxy/azure/gpt-5.4
```

The custom provider route is important because OpenCode's built-in `openai/*`
provider validates model names against its own model registry. Router-hosted
models such as `MiniMax-M3` must be registered explicitly.

---

## Permissions and web access

The runner configures OpenCode permissions for benchmark use:

| Capability | Default | Flag to allow |
| ---------- | ------- | ------------- |
| AssetOpsBench MCP tools | allowed | always enabled |
| `read`, `glob`, `grep`, `lsp` | allowed | always enabled |
| shell commands | denied | `--allow-bash` |
| file edits | denied | `--allow-edit` |
| web fetch/search | denied | `--allow-web` |
| follow-up questions | denied | not exposed |

Benchmark runs should not pass `--allow-web`. Without that flag, OpenCode's
`webfetch` and `websearch` permissions are explicitly set to `deny`.

The CLI auto-approves allowed permissions by default so unattended benchmark
runs do not block on prompts:

```text
--dangerously-skip-permissions
```

This does **not** grant access to explicitly denied tools. For example, web
access remains denied unless `--allow-web` is passed.

If you want OpenCode to ask before using allowed tools during local debugging,
pass:

```bash
--ask-permissions
```

Do not use `--ask-permissions` for batch benchmark runs; it can wait for user
input.

---

## Headless server mode

OpenCode can also run as a long-lived headless server:

```bash
opencode serve --port 4096
```

Then attach benchmark calls to it:

```bash
uv run opencode-agent --attach http://localhost:4096 \
--model-id tokenrouter/MiniMax-M3 \
"What sites are available?"
```

This can reduce startup overhead when running many scenarios. The default runner
does not require server mode; it starts OpenCode through `opencode run`.

---

## CLI flags

In addition to the [common flags](../INSTRUCTIONS.md#common-flags) (`--model-id`,
`--show-trajectory`, `--json`, `--run-id`, `--scenario-id`, ...):

| Flag | Description |
| ---- | ----------- |
| `--max-steps N` | Maximum OpenCode agentic iterations (default: 30). |
| `--agent-name NAME` | Inline OpenCode agent name (default: `assetops`). |
| `--opencode-bin PATH` | OpenCode executable path (default: `opencode`). |
| `--attach URL` | Attach to a running `opencode serve` instance. |
| `--timeout-s N` | Wall-clock timeout for `opencode run` (default: 900). |
| `--allow-bash` | Allow shell commands. Disabled by default. |
| `--allow-edit` | Allow file edits. Disabled by default. |
| `--allow-web` | Allow web fetch/search. Disabled by default. |
| `--ask-permissions` | Do not auto-approve allowed permissions. Not suitable for batch runs. |

---

## Benchmark-suite usage

The benchmark method name uses an underscore:

```bash
uv run python -m benchmark.scenario_suite_runner \
--scenario-ids benchmarks/scenario_suite/scenarios.txt \
--scenario-root /path/to/scenarios_data \
--agent_name opencode_agent \
--model-id tokenrouter/MiniMax-M3
```

The direct CLI command uses a hyphen:

```bash
uv run opencode-agent --model-id tokenrouter/MiniMax-M3 "What sites are available?"
```

For a faster smoke test, skip evaluation:

```bash
uv run python -m benchmark.scenario_suite_runner \
--scenario-ids benchmarks/scenario_suite/scenarios.txt \
--scenario-root /path/to/scenarios_data \
--agent_name opencode_agent \
--model-id tokenrouter/MiniMax-M3 \
--no-evaluate
```

---

## Validation runs

A reproducible checklist a teammate can follow to confirm a working setup:

```bash
# 0. install + entry point
uv sync
uv run opencode-agent --help
opencode --version

# 1. unit tests (no model call)
uv run pytest src/agent/opencode_agent/tests/ -q

# 2. direct smoke test
export TOKENROUTER_BASE_URL=https://api.tokenrouter.com/v1
export TOKENROUTER_API_KEY=...
uv run opencode-agent --show-trajectory \
--model-id tokenrouter/MiniMax-M3 \
"What sites are available?"

# 3. persist a trajectory
export AGENT_TRAJECTORY_DIR=$(pwd)/traces/trajectories
uv run opencode-agent --run-id opencode-smoke --scenario-id smoke \
--model-id tokenrouter/MiniMax-M3 \
"What sites are available?"

# 4. benchmark-suite dry run
uv run python -m benchmark.scenario_suite_runner \
--scenario-ids benchmarks/scenario_suite/scenarios.txt \
--scenario-root /path/to/scenarios_data \
--agent_name opencode_agent \
--model-id tokenrouter/MiniMax-M3 \
--dry-run
```

---

## Troubleshooting

| Symptom | Cause / fix |
| ------- | ----------- |
| `opencode: command not found` | Install OpenCode (`npm install -g opencode-ai`) or pass `--opencode-bin /path/to/opencode`. |
| `Model not found: openai/MiniMax-M3` | Old router mapping. Use the current runner, which maps `tokenrouter/<model>` to a custom `tokenrouter` provider instead of `openai/<model>`. |
| `TOKENROUTER_BASE_URL and TOKENROUTER_API_KEY must be set` | Set both env vars when using `tokenrouter/*`. The base URL should be OpenAI-compatible, usually ending in `/v1`. |
| `LITELLM_BASE_URL and LITELLM_API_KEY must be set` | Set both env vars when using `litellm_proxy/*`. |
| Run appears stuck after `Command: uv run opencode-agent ...` | Normal for long scenarios: `scenario_suite_runner` is quiet while OpenCode works. Check `traces/trajectories/scenario_suite/opencode_agent/` after completion. |
| Evaluator fails even though the answer text contains the right label | The answer may be too verbose for `static_json` extraction. Prefer concise answers such as `C` or JSON like `{"answer": "C"}` for strict structured tasks. |
| Web access concern | Web is denied by default. Do not pass `--allow-web` for benchmark runs. |
| OpenCode prompts for permissions in a batch run | Do not pass `--ask-permissions`; the default auto-approves allowed tools and still denies explicitly denied tools. |

---

## What was added

- `src/agent/opencode_agent/` — `runner.py`, `cli.py`, `__init__.py`, and
`tests/test_runner.py`.
- `pyproject.toml` — `opencode-agent` entry point.
- `src/benchmark/scenario_suite_runner.py` — `opencode_agent` benchmark method.

No MCP servers are modified. The runner generates OpenCode MCP/provider config
at runtime.