feat: ProgramBench reverse-engineering environment#1351

Draft

sethkarten wants to merge 41 commits into

mainfrom

feat/programbench-env

Commits on May 12, 2026

feat: add ProgramBench reverse-engineering environment

Seth
and
claude
committed
feat(programbench): add multi-language preprocess + build scripts
Seth
committed
fix(programbench): add language overrides for misclassified repos (jp2a, fasttext, brotli, halite, blake3)
Seth
committed
feat(programbench): Python-based multi-language binary builder (replaces fragile shell heredocs)
Seth
committed
fix: set GOTOOLCHAIN=local to prevent Go toolchain auto-download failures

Seth
and
claude
committed
fix: smarter Go fallback — scan subdirs for main packages

Seth
and
claude
committed
fix: strip .tar suffix before appending .tar.gz in test archive download

Seth
and
claude
committed
build: upgrade Go to 1.26.3 in programbench-toolchain Dockerfile

Seth
and
claude
committed
temp: use rust:latest for Rust sandbox until custom image is pushed

Seth
and
claude
committed
fix: cargo PATH for rust:latest + fix Dockerfile heredoc

Seth
and
claude
committed
increase default worker_heartbeat_timeout from 30s to 1800s

Seth
and
claude
committed
catch per-group exceptions in evaluate loop instead of crashing

Seth
and
claude
committed
catch per-group exceptions in evaluate loop instead of crashing

Seth
and
claude
committed
programbench: faithful replication — mini-SWE-agent + no information leakage

Seth
and
claude
committed
programbench: inject toolchain PATH into program.env for mini-swe-agent

Seth
and
claude
committed
programbench: write /etc/profile.d/pb_toolchain.sh for mini-swe-agent PATH

Seth
and
claude
committed
programbench: fix command_timeout — override 900s CLI default for mini-swe-agent

Seth
and
claude
committed
programbench: cap command_timeout at 900s (Prime Intellect API limit)

Seth
and
claude
committed
programbench: set AGENT_TIMEOUT_SECONDS to exit 60s before command_timeout hard kill

Seth
and
claude
committed
programbench: increase environment_timeout default to 600s

Seth
and
claude
committed
programbench: increase C sandbox timeout from 10 to 20 min

Seth
and
claude
committed
programbench: fix code quality issues found in review

Seth
and
claude
committed
programbench: fix P1/P2 issues from review + add lint/oo-agent hooks

Seth
and
claude
committed
programbench: fix 4 pre-merge blockers from fidelity audit

Seth
and
claude
committed
programbench: strip binary analysis artifacts from task dict; guard test exposure

Seth
and
claude
committed
programbench: require only HF_TOKEN; model key handled by runtime proxy

Seth
and
claude
committed
programbench: accept cached HF token, not just HF_TOKEN env var

Seth
and
claude
committed
add .env.example for local credentials

Seth
and
claude
committed
programbench: set agent.step_limit=1000 to match paper §4

Seth
and
claude
committed
programbench: use all test branches, matching official eval scoring

Seth
and
claude
committed
programbench: add resolved_binary metric (paper primary metric)

Seth
and
claude
committed
programbench: block internet, ban source lookup, add hash-wrap check

Seth
and
claude
committed
programbench: unified toolchain image, skip install if pre-built, expose config params

Seth
and
claude
committed
programbench: replace hardcoded team ID with PRIME_TOOLCHAIN_IMAGE env var

Seth
and
claude
committed
programbench: fix sha256sum and Go pytest timeouts

Seth
and
claude
committed
programbench: bump C and C++ pytest timeouts to 300s/120s

Seth
and
claude
committed

Commits on May 13, 2026