Problem
Autoloop programs accept iterations that break the type checker or tests because the evaluation commands don't verify correctness — they only measure the metric. For example, build-tsb-pandas-typescript-migration counts .ts files with exports, so a file with TS2322 errors still counts as "ported." PR #248 landed with type errors in hash_pandas_object.ts because of this.
The CI gate in Step 5a (push → wait for gh pr checks → accept only on green) doesn't work structurally: the PR is created by the safe_outputs job after the agent step finishes, so the agent can never gh pr checks --watch a PR that doesn't exist yet. The pending-ci status in the state file confirms this — iterations are pushed without CI verification.
Fix
Each program's evaluation command should run validity checks before reporting the metric. If checks fail, return null for the metric so the iteration is rejected.
build-tsb-pandas-typescript-migration (issue #1)
Current evaluation just counts files:
count=$(find src -name '*.ts' ... | wc -l)
echo "{\"pandas_features_ported\": ${count:-0}}"
Should also run type checking and tests:
# Type check must pass
if ! bunx tsc --noEmit 2>&1; then
echo '{"pandas_features_ported": null, "rejected_reason": "type check failed"}'
exit 0
fi
# Tests must pass
if ! bun test 2>&1; then
echo '{"pandas_features_ported": null, "rejected_reason": "tests failed"}'
exit 0
fi
# Count features
count=$(find src -name '*.ts' -not -name 'index.ts' -not -name '*.d.ts' 2>/dev/null \
| xargs grep -l 'export' 2>/dev/null | wc -l | tr -d ' ')
echo "{\"pandas_features_ported\": ${count:-0}}"
perf-comparison (issue #221)
Should verify benchmark scripts are syntactically valid before counting them.
tsb-perf-evolve (issue #189)
Already has validity checks in evaluate.sh (runs test suite before benchmarking). No change needed.
Notes
- The sandbox may not have
bun or tsc available due to network restrictions. If so, the evaluation should install them or use a fallback. Check what's available in the autoloop runner environment.
- An alternative to evaluation-time checks: add a
pre-accept hook in the autoloop workflow that runs the type checker. But evaluation-time is simpler and already has the rejection mechanism.
Problem
Autoloop programs accept iterations that break the type checker or tests because the evaluation commands don't verify correctness — they only measure the metric. For example,
build-tsb-pandas-typescript-migrationcounts.tsfiles with exports, so a file withTS2322errors still counts as "ported." PR #248 landed with type errors inhash_pandas_object.tsbecause of this.The CI gate in Step 5a (push → wait for
gh pr checks→ accept only on green) doesn't work structurally: the PR is created by thesafe_outputsjob after the agent step finishes, so the agent can nevergh pr checks --watcha PR that doesn't exist yet. Thepending-cistatus in the state file confirms this — iterations are pushed without CI verification.Fix
Each program's evaluation command should run validity checks before reporting the metric. If checks fail, return
nullfor the metric so the iteration is rejected.build-tsb-pandas-typescript-migration (issue #1)
Current evaluation just counts files:
Should also run type checking and tests:
perf-comparison (issue #221)
Should verify benchmark scripts are syntactically valid before counting them.
tsb-perf-evolve (issue #189)
Already has validity checks in
evaluate.sh(runs test suite before benchmarking). No change needed.Notes
bunortscavailable due to network restrictions. If so, the evaluation should install them or use a fallback. Check what's available in the autoloop runner environment.pre-accepthook in the autoloop workflow that runs the type checker. But evaluation-time is simpler and already has the rejection mechanism.