LoopBench points an LLM at your slow code, runs each attempt in a sandboxed loop, and evolves the file until it's faster — while guaranteeing the final patch still passes every test. LLMs happily write faster code that quietly breaks things; LoopBench makes speed and correctness a hard gate, so you only ever get a verified diff.
# BEFORE — O(2ⁿ) exponential recursion
def fibonacci(n):
if n <= 0: return 0
if n == 1: return 1
return fibonacci(n - 1) + fibonacci(n - 2)# AFTER — memoized, O(n), evolved by LoopBench
def fibonacci(n, _memo={0: 0, 1: 1}):
if n in _memo: return _memo[n]
_memo[n] = fibonacci(n - 1) + fibonacci(n - 2)
return _memo[n]On the bundled Fibonacci smoke test this lifts the score from 0.36 → 0.99 in
a few generations — every candidate re-verified against the test suite before it
counts. See examples/ for reproducible runs on primes, JSON
parsing, palindromes, and NumPy vectorization.
A recorded
loopbench runterminal cast will live here — coming soon.
Docker Desktop must be running (tests execute in an isolated sandbox).
pip install -e .
cp .env.example .env # add your LLM key (any OpenAI-compatible provider):
# GEMINI_API_KEY="..." # Groq, Gemini, OpenAI, …
# LLM_API_BASE="https://api.groq.com/openai/v1"
# LLM_MODEL="llama-3.3-70b-versatile"
# Optimize a file that already has a timing test:
loopbench run --target . --target-file examples/fibonacci_optimizer/initial_program.py --metric latency -i 5Minutes later you get a verified loopbench_output/best.patch, a validation
report, a test log, and dashboard data. The full walkthrough is in the
5-minute Quick Start.
Optimizing someone else's repo? Scaffold a job folder, edit two files, run — the target repo stays untouched:
loopbench init --job my_job # creates my_job/loopbench.yaml + test_target.py
loopbench run --config my_job/loopbench.yamlSee Defining Your Benchmark for the full config, dependencies, cost/runtime budgets, custom commands, and stdin/run mode.
- Closed-loop evolution — multi-generation optimization that learns from each failure, compounding improvements over time.
- Verified patches only — correctness is a hard gate; any failing test scores
0.0. You never receive a diff that breaks behavior. - Zero-corrupt patches — the LLM edits via full-rewrite or search/replace
blocks (
auto-routed by file size); the.patchis always computed withdifflib, so it's guaranteed to apply. - Safe sandboxing — every candidate runs in Docker with
--network=noneand a read-only mount. Bring any runner via--test-command(pytest, benchmarks, type checks, plain scripts). - Repository-aware — maps whole-repo context for the LLM, not just the single file. Third-party deps are auto-detected and installed into a cached image.
- Bounded & audit-ready — stop on a token, dollar, runtime, or iteration budget; every attempt, prompt, and metric is recorded in a SQLite audit trail.
Provider-agnostic via LLM_API_BASE / LLM_MODEL (Groq, Gemini, OpenAI, …).
Each generation runs a closed loop:
- Map — build an LLM-ready context map of the repository
- Generate — ask the LLM to improve the target file
- Apply — apply the edit in an isolated git worktree, compute a valid
.patch - Test — run the suite in a Docker sandbox (
--network=none, read-only) - Extract — parse performance metrics from test output
- Record — store the attempt in a SQLite audit database
- Select — pick the best candidate as the next baseline
For how each subsystem works — with diagrams — see
docs/architecture/.
Each folder in examples/ is a self-contained optimization you can
run directly:
| Demo | What it shows |
|---|---|
fibonacci_optimizer/ |
Hello-world: naive recursion → memoized |
prime_counter_optimizer/ |
Trial division → Sieve of Eratosthenes |
numpy_vectorize_optimizer/ |
Auto-installs NumPy, then vectorizes a Python loop |
stdin_palindrome/ |
Run mode: optimize a stdin/stdout script (no importable tests) |
loopbench run --target . --target-file examples/prime_counter_optimizer/initial_program.py --metric latency -i 5- Quick Start — clone to a verified optimization in 5 minutes
- Defining Your Benchmark — every scoring mode + full CLI flags
- Architecture — per-subsystem design with diagrams
- Contributing — dev setup, repository layout, running tests
- Dashboard — view a run's trajectory locally or on GitHub Pages
Apache-2.0 — see LICENSE. LoopBench Optimizer is a fork of OpenEvolve, also licensed under Apache-2.0.