Convert GitHub repos into Harbor tasks automatically.
Automates task creation from real open-source repos. Input any GitHub repo, and it produces a long-horizon ProgramBench-style reverse engineering task.
Every task is difficulty-calibrated and audited before export. If reward-hacking, task environment problems, or verifier design issues are found, the pipeline continues iterating through a loop until all QA and difficulty calibration is met. Task execution and grading are containerized, with required dependencies installed when each task image is built.
# Install
uv pip install programsmith
# Generate a task from a repo
programsmith create --repo d5/tengo
# Or farm multiple tasks at once
programsmith farm repos.txtuv pip install programsmithEnsure Docker is running and at least one model credential is set:
export CLAUDE_CODE_OAUTH_TOKEN=<token> # (run `claude setup-token` in your CLI)
# or
export ANTHROPIC_API_KEY=<api-key>Trials can run on any provider:
export OPENAI_API_KEY=<api-key> # codex CLI + any openai/* model
export GEMINI_API_KEY=<api-key> # gemini CLI + any gemini/* model
export ZAI_API_KEY=<api-key> # GLM models via the mini-SWE-agent harness
# Optional: one-click hosted trial + public experiment link from the dashboard
export ODDISH_API_KEY=<full-scope-key>Alias: psmith <cmd> is automatically installed
Commands:
programsmith create— Generate a task from a repoprogramsmith farm— Farm a list of repos into multiple tasksprogramsmith serve— Start the browser dashboard in the backgroundprogramsmith stop— Stop the background dashboardprogramsmith doctor— Preflight checks (Docker, credentials, disk)programsmith status— Pipeline status
programsmith create --repo <owner/repo>Runs the full pipeline (see below). Accepted tasks are exported to out/tasks/, tasks that fail
calibration but are sound are exported to out/easy/.
Re-running the same command resumes the run from wherever it parked. Ctrl-C stops active model trials, keeps completed trials, and leaves unfinished work ready to resume. The local dashboard starts automatically and opens the task page as soon as the run is created.
Options
--sha SHA— Pin a commit (default: resolve HEAD)--slug NAME— Run key (default: repo name)--smoke [HARNESS:]PROVIDER/MODEL— Initial smoke sweep agent (default:anthropic/claude-haiku-4-5on the credential-aware harness)--frontier [HARNESS:]PROVIDER/MODEL— Difficulty calibration agent (default:anthropic/claude-opus-4-8; e.g.codex:openai/gpt-5.5,gemini-cli:gemini/gemini-3.1-pro-preview,mini-swe:zai/glm-5.2)--config FILE.json/--preset NAME— Full RunConfig (agents + per-model bands)--brief TEXT— Steer the task generation scope (eg "port the FFT subsystem...")--review— Pause at the two human gates (scope pick, final QA) instead of auto--draft— Export after Static CI with no sweeps or calibration--yes— Skip the cost preview confirmation--no-open-dashboard— Do not open the task page in a browser--runs-dir PATH— Choose the directory for runs (default:.programsmith/runs)--allow-copyleft— Allow copyleft-licensed sources
programsmith farm repos.txtStreams through a repo list (one owner/name[@sha] per line) and drives every run to
completion in the foreground. Re-running the command picks
up existing runs where they stopped. A curated starter list of known-good repos is included
(pb10-repos.txt).
Options
- All flags from
create:--smoke,--frontier,--config,--preset,--review,--yes,--runs-dir, shared for all runs --no-drive— Only create + ingest the runs (drive them later withserve)--no-prune— Skip thedocker image prunehygiene between completed runs
programsmith serve
# The dashboard runs in the background and survives Ctrl-C / closing this shell.
programsmith stopServes the local dashboard at http://localhost:8765. Each exported task has a direct download
button and can launch one low-priority Oddish trial, show its agent trajectory, and return a public
experiment link. Build diagnostics, files, sweep results, and optional review gates remain
available without dominating the task page. Add a full-scope Oddish API key in Settings to enable
the hosted run; Oddish free-plan limits apply. Evaluation sweeps remain parked unless
serve --spend is used.
programsmith serve returns after the dashboard is healthy; programsmith stop is the explicit
shutdown command.
What repos work well
Best sources: small-to-medium CLI tools with deterministic stdin/stdout behavior (formatters, converters, parsers, interpreters, compression tools, query tools).
A good source repo:
- Builds cleanly in a container (Go, Rust, C, C++, etc)
- Has a CLI surface with deterministic, byte-reproducible output
- Is permissively licensed (verified in pipeline)
- Is not already saturated by frontier models
The pipeline also auto-rejects repos already used by public ProgramBench-style datasets.
Pipeline details
ProgramSmith runs a fixed DAG of deterministic gates. LLM work is quarantined to synthesis cells whose JSON output is schema-validated before passing any gate. The orchestrator routes only on gate verdicts.
- Ingest + Lock — clone, pin SHA, check license and overlap
- Task Matrix — pick the task scope
- Oracle + Goldens — build the reference oracle and generate public + held-out golden I/O cases (including adversarial cases)
- Create — make the task: repo, instruction, verifier, Dockerfile
- Sanity Check — check oracle scores 1, nop (empty solution) scores 0
- Static Checks — anti-cheat check suite (closed internet, asset encryption, no reviewer-visible goldens, reward format, etc)
- Smoke Sweep — N smoke-model trials; 100% pass ⇒ saturated ⇒ ease or drop
- Calibrate — band check; out-of-band tasks are hardened/eased via patches
- Audit Probe — adversarial agent looks for reward hacks, exploits are fixed
- Frontier Sweep — N frontier-model trials; target pass@1 in [1/3, 2/3] or whatever is specified
- QA Gate — good-failure analysis on 0-pass tasks, accepted tasks are exported