Skip to content

Reject perf-comparison iterations with invalid benchmark scripts - #254

Merged
mrjf merged 2 commits into
mainfrom
copilot/add-type-checking-tests-autoloop
Apr 30, 2026
Merged

Reject perf-comparison iterations with invalid benchmark scripts#254
mrjf merged 2 commits into
mainfrom
copilot/add-type-checking-tests-autoloop

Conversation

Copilot AI commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

Autoloop evaluation commands only measured their metric and never verified correctness, so iterations that broke benchmark scripts (or, in the sibling pandas-migration program, the type checker / tests) were silently accepted. This updates the perf-comparison evaluator to gate the metric on benchmark-script validity.

Changes

  • .autoloop/programs/perf-comparison/program.md — evaluation block now runs validity checks before counting:
    • bun build each benchmarks/tsb/bench_*.ts (catches parse, transpile, and import-resolution errors)
    • python3 -m py_compile each benchmarks/pandas/bench_*.py
    • On any failure, or if bun / python3 is missing, emit {"benchmarked_functions": null, "rejected_reason": "..."} so the autoloop runner rejects the iteration.
for f in benchmarks/tsb/bench_*.ts; do
  [ -e "$f" ] || break
  if ! bun build "$f" --outdir=/tmp/perf-comparison-bench-check >/dev/null 2>&1; then
    echo "{\"benchmarked_functions\": null, \"rejected_reason\": \"invalid TypeScript benchmark: $f\"}"
    exit 0
  fi
done

Running the new evaluator against the current repo immediately surfaces 5 pre-existing broken TS benchmarks (e.g. bench_dataframe_mask.ts imports a non-existent dataFrameMask export) and a broken Python benchmark — exactly the regression class the gate is meant to catch. Those scripts are left as-is (out of scope) and will now correctly block iterations until fixed.

Out of scope

  • build-tsb-pandas-typescript-migration — its program definition lives in Issue Build tsb: pandas → TypeScript migration #1, not in a repo file. AGENTS.md forbids in-repo modification and no tool here can edit issue bodies. A maintainer should add the bunx tsc --noEmit + bun test gate to that issue body per the proposal in the originating issue.
  • tsb-perf-evolve — already validates via bun test …/series.sortValues.test.ts in evaluate.sh; no change needed.

Copilot AI changed the title [WIP] Add type checking and tests to autoloop evaluation commands Reject perf-comparison iterations with invalid benchmark scripts Apr 30, 2026
Copilot finished work on behalf of mrjf April 30, 2026 16:19
Copilot AI requested a review from mrjf April 30, 2026 16:19
@mrjf
mrjf marked this pull request as ready for review April 30, 2026 21:27
@mrjf
mrjf merged commit 8fa39cb into main Apr 30, 2026
18 checks passed
@mrjf
mrjf deleted the copilot/add-type-checking-tests-autoloop branch April 30, 2026 21:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add type checking and tests to autoloop evaluation commands

2 participants