Skip to content

defining-benchmarks.md

Latest commit

 

History

History
687 lines (518 loc) · 25.6 KB

defining-benchmarks.md

File metadata and controls

687 lines (518 loc) · 25.6 KB
Raw

Defining Your Benchmark

A "benchmark" in LoopBench is two things working together:

  1. A correctness gate — your tests pass/fail. Any failure forces the score to 0.0, so a candidate that breaks behavior is always rejected.
  2. A metric to optimize — a number (latency, memory, accuracy, …) that LoopBench tries to improve, generation after generation.

These combine into a single combined_score that drives evolution:

combined_score = correctness × metric_score

There are four ways to define this, from fastest to most flexible (A–D below). Pick one — then see Optimizing an external repo to package it as a config job for someone else's repo.

Option A — Hero mode (fastest): tests + a speed marker

Best when you just want to point LoopBench at a file and go. The benchmark is your repo's own pytest suite. You need two things in a test_*.py file:

  • Assertions that verify correctness.
  • One test that measures the hot path and prints a marker line: LOOPBENCH_SPEED_MS=<number>.

Full flow

# 0. Docker must be running
docker info

# 1. Your project layout — the test lives next to the file being optimized
#    my_repo/
#    ├── slow_module.py        <- file to optimize
#    └── test_slow_module.py   <- correctness + speed marker

Write the test so it emits the speed marker:

# test_slow_module.py
import importlib.util, os, time, types
import pytest

_PATH = os.environ["LOOPBENCH_PROGRAM_PATH"]  # set by LoopBench

def _load():
    spec = importlib.util.spec_from_file_location("evolved", _PATH)
    m = importlib.util.module_from_spec(spec); spec.loader.exec_module(m)
    return m

@pytest.fixture(scope="session")
def prog(): return _load()

# ── correctness gate ──
def test_correct(prog):
    assert prog.solve(10) == 55

# ── the metric being optimized ──
def test_speed(prog):
    start = time.perf_counter()
    prog.solve(50_000)
    elapsed_ms = (time.perf_counter() - start) * 1000
    print(f"\nLOOPBENCH_SPEED_MS={elapsed_ms:.4f}")   # <- LoopBench reads this
    assert elapsed_ms < 5000                          # hard safety limit

Run the optimization:

loopbench run \
  --target . \
  --target-file slow_module.py \
  --metric latency \
  -i 5

# Optional: override the auto-detected test command
loopbench run --target . --target-file slow_module.py \
  --test-command "pytest test_slow_module.py -s -q" -i 5

Where the score is computed

The scoring formula lives in sandbox/entrypoint.sh:

correctness    = 1.0 if all_passed else 0.0
speed_score    = exp(-speed_ms / 150.0)     # tune the 150.0 for your latency scale
combined_score = correctness * speed_score

If your latencies are much larger (e.g. thousands of ms), raise the 150.0 decay constant so scores spread meaningfully across your range.

Template you can copy: examples/prime_counter_optimizer/test_prime_counter.py

Option B — Custom scoring: combine multiple signals

Best when "better" isn't a single timing — e.g. you want to weight accuracy vs. speed vs. memory. You don't need a separate evaluator.py — your test file is the evaluator. Compute whatever you want inside the test and expose it two ways:

  • Do the weighting yourself and print one number as LOOPBENCH_SPEED_MS (Option A) — e.g. score = time_ms / accuracy, so lower time and higher accuracy both help.
  • Or print several named numbers and choose which to optimize with a regex metric (Option C) plus --metric <name>.

Correctness stays the hard gate: any failing test scores 0.0.

Advanced (rarely needed). A fully custom evaluate(program_path) -> EvaluationResult scorer is also supported through the separate OpenEvolve optimizer engine (optimizer run --config ...).

This is not "the smart part" — the intelligence (the LLM that writes the improvements + the evaluator-driven loop that keeps the best and learns from failures) is identical in both. The only difference is the search strategy: loopbench run uses greedy/beam selection, while the optimizer engine adds MAP-Elites / island populations for broader exploration on very hard, open-ended problems. For "make this faster and keep tests green," the default flow is just as capable (and cheaper).

Option C — Custom regex metric: parse numbers from existing test output

Best when your tests already print performance numbers in their own format and you don't want to adopt the LOOPBENCH_SPEED_MS convention. LoopBench extracts the metric with a regex via its MetricParser.

Add a metrics section to your config:

metrics:
  patterns:
    execution_time: 'Mean:\s*([\d.]+)\s*seconds'   # captures the number in group 1
    throughput:     'ops/sec:\s*([\d.]+)'
  success_threshold: 0.10        # min improvement to count as a win

Set which captured metric to optimize via metric.name in the config (e.g. throughput), then run:

loopbench run --config my_project/loopbench.yaml

LoopBench runs your tests, greps each pattern out of stdout/stderr, and optimizes the metric you named (falling back to combined_score).

Option D — Run mode: stdin/stdout scripts (no importable tests)

Best for scripts that read stdin and write stdout at module top level — competitive-programming solutions, CLI tools, filters. The default harness imports the target, which crashes such scripts (they call input() on import). Run mode instead executes the target as a subprocess, feeds it stdin, and compares stdout against expected output.

Provide a JSON file of I/O cases:

[
  {"name": "sample",  "input": "2\nab\nbabba\n", "output": "NO\nYES"},
  {"name": "big_no",  "input": "1\nabcabc...\n", "output": "NO"}
]

Then point LoopBench at the script with --io-tests:

loopbench run \
  --target . \
  --target-file path/to/solution.py \
  --io-tests path/to/io_tests.json \
  -i 6

LoopBench auto-generates a pytest harness that runs python solution.py per case, checks the output, and times the heaviest case for the speed score. You can also drop the cases in a conventional file next to the target (<stem>.io.json or io_tests.json) and omit --io-tests — it's auto-detected.

  • Correctness = every case's stdout matches (compared line-by-line, trailing whitespace ignored).
  • The metric = wall-clock time of the largest input.

Template you can copy: examples/stdin_palindrome/ (solution.py + io_tests.json). Verified end-to-end: a naive O(n³) stdin solver was evolved to O(n), all cases green, on a script that can't even be imported.

Constraints & cost budget

LoopBench is cost-bounded: the loop stops early when a token or dollar budget is reached (in addition to --iterations). This applies to hero mode (--target ...).

# Stop after 50k total LLM tokens (works with any provider — tokens are
# reported by the API and always enforceable)
loopbench run --target . --target-file src/hot.py --metric latency --max-tokens 50000

# Stop after an estimated $0.25 spend (requires pricing, see below)
loopbench run --target . --target-file src/hot.py --metric latency --max-cost 0.25

# Stop after 300 seconds of wall-clock time
loopbench run --target . --target-file src/hot.py --metric latency --max-runtime 300

Or declare them in loopbench.yaml under constraints (CLI flags override):

constraints:
  max_iterations: 20
  max_tokens_total: 50000          # hard token budget
  max_token_cost_usd: 0.25         # dollar budget (needs pricing below)
  max_runtime_seconds: 300         # wall-clock deadline for the whole run
  usd_per_1k_prompt: 0.00059       # your provider's input price per 1k tokens
  usd_per_1k_completion: 0.00079   # output price per 1k tokens

The other two constraints from the spec are always on and need no config: the sandbox runs with --network=none (no external network) and mounts your code read-only (no unsafe file writes).

Search strategy (auto-tuning)

loopbench run chooses which prior candidate each new generation builds on. By default this is auto: it starts greedy (cheapest) and escalates only when the run stops improving — no extra LLM calls, fully deterministic.

stall (generations since last improvement) Behavior
< restart_patience (default 2) greedy — build on the best so far
restart_patience … diversify_patience (2–4) restart — revert to the original baseline for a fresh path
≥ diversify_patience (default 4) diversify — rotate through the top-K candidates

You rarely need to touch this. To pin a strategy, use --strategy or a search: block in loopbench.yaml (CLI overrides config):

loopbench run --target . --target-file src/hot.py --strategy random_restart
search:
  strategy: auto            # auto (default) | greedy | beam | random_restart
  restart_patience: 2       # auto: greedy → restart threshold
  diversify_patience: 4     # auto: restart → diversify threshold
  beam_width: 3             # auto (diversify tier) and beam
  restart_interval: 20      # random_restart only

Escalation only steers exploration — the run always reports the highest-scoring candidate, so auto never does worse than plain greedy. For the design and per-strategy details, see Architecture → Search Strategy.

Optimizing an external repo (config-driven)

To optimize a file in someone else's repo, you write a small job folder in your own workspace — the same structure as the examples — and point its loopbench.yaml at the external repo. Scaffold it in one command so you only edit configuration:

loopbench init --job my_job     # creates my_job/loopbench.yaml + my_job/test_target.py
``` You never edit files inside the target
repo; LoopBench clones it, injects your evaluator into the sandbox, and
optimizes the named file.

my_job/ ├── loopbench.yaml # points at the external repo + file to optimize └── test_target.py # the evaluator/test (correctness gate + LOOPBENCH_SPEED_MS)


`loopbench.yaml`:

```yaml
target:
  repo: https://github.com/OmkarPathak/Python-Programs   # cloned automatically
  file: MachineLearning/gradient_descent.py              # the file to optimize
  evaluator: test_target.py                              # local (in this job dir)

sandbox:
  command: "pytest test_target.py -v -s -q"
  pip: ["numpy", "matplotlib"]                           # installed in the sandbox

metric:
  name: "combined_score"
  threshold: 0.90

constraints:
  max_iterations: 20
  max_tokens_total: 200000

Then run:

loopbench run --config my_job/loopbench.yaml

test_target.py loads each candidate from LOOPBENCH_PROGRAM_PATH, asserts correctness, and prints LOOPBENCH_SPEED_MS — exactly like Option A/B above. The target repo stays untouched; everything you author lives in my_job/.

A local file you own keeps using target.program (evaluator-first controller); target.repo + target.file is the external-repo path (clone + sandbox).

Third-party dependencies (numpy, pandas, …)

Real code imports packages that aren't in the base sandbox. LoopBench detects them and installs them into a cached, per-dependency-set image layered on the base — the install is the only networked step; the scored run stays --network=none. Detection priority (authoritative first):

  1. --pip "numpy scipy" (explicit; also settable as sandbox.pip in config)
  2. a requirements.txt at the repo root
  3. pyproject.toml dependencies (PEP 621 or Poetry)
  4. imports scanned across every .py file in the repo — a best-effort fallback (import names mapped to PyPI names; stdlib/local modules filtered)
# Auto-detected from the repo's requirements.txt / pyproject.toml / imports:
loopbench run --target . --target-file src/model.py --metric latency

# Or pin them explicitly (fast, deterministic — recommended for real projects):
loopbench run --target . --target-file src/model.py --pip "numpy scipy"

The run prints which source the dependencies came from, so you can gauge confidence. The first run with a new dependency set builds the image (a minute or two); later runs reuse it.

Reliability note. Import-scanning is a convenience, not a guarantee — it can't resolve every import→PyPI name (e.g. cv2→opencv-python), misses dynamic/optional imports, and infers no versions. For large or real projects, prefer a declared source: a requirements.txt/pyproject.toml in the repo, or an explicit --pip / sandbox.pip. LoopBench always trusts those over the import scan.

Custom sandbox commands (any test/benchmark runner)

By default the sandbox runs pytest. To use a different command — a benchmark harness, a type checker, or a plain script — pass --test-command (hero mode) or set sandbox.command (config mode):

loopbench run --target . --target-file src/hot.py \
  --test-command "python benchmark.py"

For a pytest command, correctness comes from the pass/fail report. For any other command, correctness is the command's exit code (0 = pass). In both cases, print a LOOPBENCH_SPEED_MS=<number> line to feed the speed score.

Token counts come from the provider's usage field (OpenAI, Groq, and Google AI Studio all report it). The dollar estimate needs the two pricing fields — if they're 0, the USD budget is inactive but the token budget still works. Every generation's token/cost delta is written to the run's audit log, and the run summary reports total tokens, estimated cost, and whether the budget stopped it.

Where to find your pricing

Set usd_per_1k_prompt / usd_per_1k_completion to your provider's current per-1,000-token rates (pricing changes often — always check the provider page):

  • Groq — groq.com/pricing
  • OpenAI — openai.com/api/pricing
  • Google Gemini — ai.google.dev/pricing

Example: if a provider charges $0.59 per million input tokens, that is 0.59 / 1000 = 0.00059 per 1k, so usd_per_1k_prompt: 0.00059.

Worked example

A ready-to-use budget block ships in examples/fibonacci_optimizer/loopbench.yaml. Run the fibonacci demo with a small token cap:

loopbench run \
  --target . \
  --target-file examples/fibonacci_optimizer/initial_program.py \
  --metric latency \
  -i 5 \
  --max-tokens 1

Because the baseline uses no LLM tokens, generation 1 runs, then the loop stops at the budget gate before generation 2. The summary reports what was spent:

  Total Generations: 1
  ...
  Tokens used    : 896 (1 API calls)
------------------------------------------------------------

Swap --max-tokens 1 for a realistic value (e.g. --max-tokens 50000) or use --max-cost 0.25 once pricing is set in the config.

Measuring speed reliably (statistical speed gate)

A single timing is noisy. The same code can measure 5% faster or slower run to run because of CPU scheduling, cache state, and background load. Left unchecked, that noise lets LoopBench "accept" a candidate that isn't actually faster — it just got a lucky sample. The statistical speed gate measures each candidate several times and only accepts a win that clears the noise.

It is off by default in the sense that the defaults reproduce the original single-shot behavior: repeats=1 measures once and accepts any median improvement, exactly as before. Turn it on by asking for more repeats.

Measuring each candidate more than once

Set sandbox.repeats in loopbench.yaml to the number of times the speed workload is measured per candidate:

sandbox:
  repeats: 5        # measure each candidate 5 times (default: 1)
  • repeats: 1 (default) — single-shot measurement. Identical to the original behavior; nothing else in this section changes the outcome.
  • repeats >= 3 — the first run is discarded as a warm-up (it pays cold-cache and JIT/import costs that don't reflect steady state), and the remaining runs are kept.
  • repeats: 5 is the recommended value for noise-robust measurement: after dropping the warm-up you keep four samples, enough for a stable median and a meaningful spread without paying for many extra runs.

Repeated measurement only re-runs the sandboxed workload — it costs no extra LLM tokens.

The acceptance rule (median-over-K + noise gate)

With repeats configured, a candidate becomes the new best only if both conditions hold:

  • (a) Real effect: the relative median improvement clears metric.min_effect — (median_base − median_cand) / median_base >= min_effect.
  • (b) Above the noise: the candidate's median, padded by the larger of the two spreads, still beats the baseline median — median_cand + max(stddev_cand, stddev_base) < median_base.
metric:
  min_effect: 0.03   # require a 3% relative median speedup (default: 0.03)

In plain language: condition (a) rejects wins too small to matter, and condition (b) rejects wins that are within measurement noise. If the improvement is smaller than how much the numbers jitter between runs, we don't trust it, so we don't accept it. With repeats=1 both spreads are 0, so condition (b) collapses to "is the candidate's median lower?" and the gate behaves like the original single-shot comparison.

New score.json fields

Each scored candidate now records the full speed distribution, not just one number:

Field Meaning
speed_ms The median of the kept runs (was a single timing before). Kept for backward compatibility.
speed_ms_median Median of the kept per-run timings (ms).
speed_ms_mean Mean of the kept timings (ms).
speed_ms_stddev Sample standard deviation of the kept timings (ms); 0.0 when only one run is kept.
speed_ms_samples The kept per-run timings (warm-up already dropped).
runs How many runs were kept.

Because speed_ms is now the median, older consumers that read speed_ms continue to work unchanged.

Revalidating the winner

Even with repeated measurement, the candidate that wins the loop was chosen partly because it measured well during the loop. To guard against a winner that got a favorable sample, LoopBench re-runs the winning candidate after the loop and checks that the speed gain still holds. This is on by default.

  • Revalidation re-applies the winner's patch and re-measures it M times in the sandbox (default M = 7). It builds no prompts and makes no LLM calls, so it adds no token cost.
  • The run is marked successful only if the re-measured distribution still clears the same speed gate (conditions (a) and (b) above) against the baseline. If the gain no longer holds, the status is downgraded to revalidation_failed — the code and patch are still reported, but the claimed speedup didn't survive re-measurement.

Control it from the CLI:

# Revalidation is ON by default — nothing to do to enable it.
loopbench run --target . --target-file src/hot.py --metric latency

# Re-measure the winner 15× instead of the default 7
loopbench run --target . --target-file src/hot.py --metric latency \
  --revalidate-runs 15

# Skip revalidation entirely
loopbench run --target . --target-file src/hot.py --metric latency \
  --no-revalidate

--no-revalidate turns the final check off; --revalidate-runs N sets how many times the winner is re-measured (default 7, ignored when --no-revalidate is set).

Running non-Python toolchains (any language)

Everything above assumes the default sandbox — a Python image running pytest. But the correctness/speed contract is language-agnostic: a candidate passes if a command exits 0, and its speed is whatever it prints as LOOPBENCH_SPEED_MS. Two config keys let you point that contract at a Node, Go, Rust, or any other toolchain by swapping the base image.

sandbox.image — a custom base image

sandbox.image (default null → the default LoopBench Python sandbox) sets the base Docker image the candidate runs and is scored in. Point it at any image that has your toolchain:

sandbox:
  image: node:20-alpine        # or golang:1.22-alpine, rust:1-slim, …

Leaving it unset (or blank) keeps today's behavior exactly — the Python sandbox, byte-for-byte. When you do set it, LoopBench builds a small derived image on top of your base and layers in its own entrypoint + scorer automatically (see How scoring works without Python below), so you only supply the base image, any build steps, and a command.

sandbox.setup — one-time build steps

sandbox.setup (default [] → none) is an ordered list of shell commands run once at image-build time, on top of the base, before any candidate runs. Use it to install dependencies or compile a project:

sandbox:
  image: node:20-alpine
  setup:
    - npm ci
    - npm run build

A single string is also accepted (setup: "npm ci") and treated as one step. Order is preserved and steps are not de-duplicated — they are literal RUN lines in the derived image, so npm ci then npm run build run in that order.

The derived image is cached by a hash of the base image, the setup steps, and any pip packages, so repeated runs reuse it instead of rebuilding. As with the Python dependency layer, the build is the only step with network access — the scored run still executes with --network=none.

pip is never forced onto a custom base. sandbox.pip packages are layered only when you actually declare them, so an image without Python (or without pip) works fine.

How scoring works without Python

For a pytest sandbox.command (or --test-command), scoring is unchanged — the structured pass/fail report drives correctness. For any other command, LoopBench uses a toolchain-agnostic scorer written in shell + awk with no python3 dependency, so it runs on images that have no Python at all:

  • Correctness = the command's exit code (0 = pass, nonzero = fail).
  • Speed = the median of the LOOPBENCH_SPEED_MS markers the command prints (across repeats, warm-up dropped when repeats >= 3).

The formula, rounding, and score.json fields are identical to the Python scorer for the same inputs — the only difference is the implementation language. When you use a custom image, LoopBench copies its entrypoint and this generic scorer into the derived image and sets the entrypoint for you; nothing about the scoring contract changes.

Limitation — correctness is exit-code based. Outside the pytest path there is no structured test report, so a non-pytest command must exit nonzero when it fails and 0 only when it truly passes. If your command always exits 0 (e.g. a runner that prints "FAILED" but returns success), correctness will read as passing. Make the command itself assert and exit nonzero on failure — or run your tests through pytest to get report-based correctness.

Worked example — a Node target

Optimize a Node solution. The target prints LOOPBENCH_SPEED_MS=<ms> for the hot path and exits 0 when the result is correct (nonzero otherwise):

// solution.js
function solve(n) { /* the hot path being optimized */ }

const t0 = process.hrtime.bigint();
const result = solve(50000);
const elapsedMs = Number(process.hrtime.bigint() - t0) / 1e6;

console.log(`LOOPBENCH_SPEED_MS=${elapsedMs}`);   // <- LoopBench reads this
process.exit(result === 1666416667 ? 0 : 1);       // exit 0 = pass, nonzero = fail
# loopbench.yaml
target:
  file: solution.js

sandbox:
  image: node:20-alpine
  setup: ["npm ci"]            # omit if the target has no dependencies
  command: "node solution.js"  # non-pytest → exit-code correctness + speed marker

metric:
  name: "combined_score"
loopbench run --config loopbench.yaml

LoopBench builds node:20-alpine + npm ci once, then runs node solution.js per candidate with --network=none, scoring correctness from the exit code and speed from the printed marker — the same combined_score = correctness × speed you get in Python mode. See Custom sandbox commands for the command key and Third-party dependencies for how the caching layer relates to sandbox.pip.

Inspecting the benchmark result

Every run writes artifacts to loopbench_output/ (hero mode) or your configured output dir:

# The winning diff
cat loopbench_output/best.patch

# Before/after metrics and patch status
cat loopbench_output/report/validation_report.md

# Proof the winning candidate kept all tests passing
cat loopbench_output/test_log.txt

The run also writes docs/data.json; view the trajectory on the dashboard:

python -m http.server 8080 --directory docs   # then open http://localhost:8080

Which option should I use?

You want… Use Where you set the benchmark
Point-and-go on a file/repo A test_*.py (LOOPBENCH_SPEED_MS) + sandbox/entrypoint.sh formula
Combine multiple signals (accuracy + speed + memory) B compute one score in your test_*.py, or a named metric via C
Reuse existing perf output C metrics.patterns regex in the config
Optimize a stdin/stdout script D --io-tests JSON of input/output cases

All modes are cost-bounded — see Constraints & cost budget to cap tokens or dollars.

See the Quick Start for the end-to-end 5-minute walkthrough.