claude-autoresearch

Bench-driven optimization loops for any scalar metric, powered by Claude Code.

Run a closed loop where an AI agent hypothesizes a change, edits scope-bounded files, measures impact, keeps the win or reverts, and logs everything — until time runs out.

Inspired by karpathy/autoresearch (ML training loop) and davebcn87/pi-autoresearch (generalized to any metric). This repo is the Claude Code-native implementation: drop the two skill files into any project's .claude/skills/ and you're ready.

---

What it does

Given a single scalar metric to optimize (test wall-time, bundle size, latency, memory, anything that fits in one number), the agent:

loop until budget runs out:
  1. Read the goal + history (autoresearch.jsonl)
  2. Hypothesize ONE change to try
  3. Edit ONE OR MORE files within the declared scope
  4. Run bench script → capture the metric
  5. Run invariants script → pass/fail (tests, lint, typecheck)
  6. If improvement > noise threshold AND invariants pass:
     → KEEP (git commit)
  7. Else:
     → REVERT (git checkout)
  8. Append to autoresearch.jsonl

The agent decides hypotheses autonomously based on the goal, scope, and prior iterations. You read the log, review the kept commits, accept or revert.

When to use

• Yes: a clear scalar to push, < 10 min bench, tractable scope (1-10 files), invariants you can express as a shell exit code.
• No: vague goal, multi-objective optimization, manual user input mid-loop, scope spanning the whole codebase.

Quick start

1. Pick a target metric

Examples that work well:

• Wall-time of a test suite
• Bundle size of a frontend build
• p95 latency of an HTTP endpoint
• Lighthouse score
• Binary size of a release artifact

2. Install the skills

Copy the skills/autoresearch-loop/ folder into your project's .claude/skills/ directory.

Optionally also copy skills/autoresearch-scout/ if you have an Nx monorepo + GitHub Actions and want help identifying which target to optimize.

3. Create the harness folder

For each metric you want to optimize, create a folder containing three files:

my-optimization/
├── autoresearch.md         # Goal + scope + constraints
├── autoresearch.sh         # Bench: prints ONE number on stdout
└── autoresearch.checks.sh  # Invariants: exit 0 if all checks pass

Browse examples/ for concrete templates by tech stack.

4. Run the loop

In Claude Code, run:

/autoresearch-loop start

The skill takes over: baselines, iterates, commits or reverts. Watch the JSONL log update in real time.

5. Review and ship

When the loop finishes (budget, plateau, or you stop it), it writes a summary:

autoresearch-summary.md

The kept commits are on the current branch — review them as you would any PR.

---

The two skills

autoresearch-loop ⭐ (the core)

Stack-agnostic. Reads the three harness files, runs the loop, manages git, logs to JSONL. Use this on any repo, any language.

autoresearch-scout (optional, Nx + GitHub Actions specific)

For monorepos with many targets, automates the "which target should I optimize?" question:

• Inventories all project.json build/test/lint targets
• Mines GitHub Actions run history via gh CLI
• Computes per-target stats (mean, p90, runs/week, cost/week)
• Scores each candidate by impact × tractability
• Outputs a ranked report + draft harness for the top 3 candidates

Skip this skill if your repo is not an Nx monorepo or doesn't run on GitHub Actions. You can still use the loop with manually-chosen targets.

---

How it stays honest

A few safety rails baked into the skill spec:

• Single-file edits per iteration (when possible) — smaller diffs are easier to revert and explain.
• Cite the hypothesis BEFORE editing — every commit message includes the rationale.
• Never modify the harness files during the loop (or the bench becomes untrustworthy).
• Never skip tests, lower coverage, or weaken assertions to "improve" the metric. That's the cardinal cheat — flag and revert.
• MAD-based noise scoring (Median Absolute Deviation) — distinguishes real wins from variance.

---

Examples

Stack	Metric	Folder
Node.js + Jest	Test wall-time	examples/nodejs-test-speedup/
Python + pytest	Test wall-time	examples/python-pytest-speedup/
Go	Bench latency	examples/go-bench-latency/

Each example folder has its own README explaining the harness contents and expected output.

---

Prior art

• karpathy/autoresearch — the original. Pure-ML focus: a script edits train.py, runs a 5-min training, measures val_bpb, keeps the improvement. About 12 experiments per GPU hour.
• davebcn87/pi-autoresearch — generalizes the loop to any scalar metric, runs on the pi.dev platform.
• Shopify's adoption write-up — applied to dev-loop optimizations: Polaris VRT −65%, unit tests 300× faster, Liquid −53%, React mount +20%.

This repo brings the same loop to Claude Code, with no external SaaS dependency. Everything runs locally against your repo's git history.

---

License

MIT — see LICENSE.