Skip to content

Reject perf-comparison iterations with invalid benchmark scripts #254

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mrjf merged 2 commits into from
Apr 30, 2026
Merged

Reject perf-comparison iterations with invalid benchmark scripts #254

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6012dce
Initial plan
Copilot Apr 30, 2026
ed83ab2
perf-comparison: validate benchmark scripts before counting them
Copilot Apr 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 39 additions & 5 deletions .autoloop/programs/perf-comparison/program.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,18 +50,50 @@ Do NOT modify:

## Evaluation

The evaluation block runs validity checks **before** counting benchmark pairs.
If any benchmark script is syntactically invalid (or required tooling is
missing), the metric is reported as `null` so the iteration is rejected
rather than silently accepted with broken benchmarks.

```bash
# Set up Python environment if needed
# Set up Python environment if needed.
if ! command -v python3 &>/dev/null; then
echo "Python3 not found, skipping"
echo '{"benchmarked_functions": null, "rejected_reason": "python3 not available"}'
exit 0
fi
pip3 install pandas --quiet 2>/dev/null || true

# Count the number of benchmark pairs (functions with both TS and Python benchmarks)
# Validity: every TypeScript benchmark must transpile cleanly.
# `bun build` parses, type-aware-transpiles, and resolves imports — any of
# those failing means the benchmark would fail at run time. We discard the
# build output; we only care about the exit status.
if command -v bun &>/dev/null; then
for f in benchmarks/tsb/bench_*.ts; do
[ -e "$f" ] || break
if ! bun build "$f" --outdir=/tmp/perf-comparison-bench-check >/dev/null 2>&1; then
echo "{\"benchmarked_functions\": null, \"rejected_reason\": \"invalid TypeScript benchmark: $f\"}"
exit 0
fi
done
else
echo '{"benchmarked_functions": null, "rejected_reason": "bun not available"}'
exit 0
fi

# Validity: every Python benchmark must compile (parse) cleanly.
for f in benchmarks/pandas/bench_*.py; do
[ -e "$f" ] || break
if ! python3 -m py_compile "$f" 2>/dev/null; then
echo "{\"benchmarked_functions\": null, \"rejected_reason\": \"invalid Python benchmark: $f\"}"
exit 0
fi
done

# Count the number of benchmark pairs (functions with both TS and Python benchmarks).
ts_benchmarks=$(ls benchmarks/tsb/bench_*.ts 2>/dev/null | wc -l | tr -d ' ')
py_benchmarks=$(ls benchmarks/pandas/bench_*.py 2>/dev/null | wc -l | tr -d ' ')

# The metric is the minimum of the two (both must exist for a complete benchmark)
# The metric is the minimum of the two (both must exist for a complete benchmark).
if [ "$ts_benchmarks" -lt "$py_benchmarks" ]; then
count=$ts_benchmarks
else
Expand All @@ -71,4 +103,6 @@ fi
echo "{\"benchmarked_functions\": ${count:-0}}"
```

The metric is `benchmarked_functions`. **Higher is better.**
The metric is `benchmarked_functions`. **Higher is better.** When validity
checks fail the metric is `null`, which the autoloop runner treats as a
rejected iteration.
Loading