Skip to content

Add type checking and tests to autoloop evaluation commands #253

Description

@mrjf

Problem

Autoloop programs accept iterations that break the type checker or tests because the evaluation commands don't verify correctness — they only measure the metric. For example, build-tsb-pandas-typescript-migration counts .ts files with exports, so a file with TS2322 errors still counts as "ported." PR #248 landed with type errors in hash_pandas_object.ts because of this.

The CI gate in Step 5a (push → wait for gh pr checks → accept only on green) doesn't work structurally: the PR is created by the safe_outputs job after the agent step finishes, so the agent can never gh pr checks --watch a PR that doesn't exist yet. The pending-ci status in the state file confirms this — iterations are pushed without CI verification.

Fix

Each program's evaluation command should run validity checks before reporting the metric. If checks fail, return null for the metric so the iteration is rejected.

build-tsb-pandas-typescript-migration (issue #1)

Current evaluation just counts files:

count=$(find src -name '*.ts' ... | wc -l)
echo "{\"pandas_features_ported\": ${count:-0}}"

Should also run type checking and tests:

# Type check must pass
if ! bunx tsc --noEmit 2>&1; then
  echo '{"pandas_features_ported": null, "rejected_reason": "type check failed"}'
  exit 0
fi

# Tests must pass
if ! bun test 2>&1; then
  echo '{"pandas_features_ported": null, "rejected_reason": "tests failed"}'
  exit 0
fi

# Count features
count=$(find src -name '*.ts' -not -name 'index.ts' -not -name '*.d.ts' 2>/dev/null \
  | xargs grep -l 'export' 2>/dev/null | wc -l | tr -d ' ')
echo "{\"pandas_features_ported\": ${count:-0}}"

perf-comparison (issue #221)

Should verify benchmark scripts are syntactically valid before counting them.

tsb-perf-evolve (issue #189)

Already has validity checks in evaluate.sh (runs test suite before benchmarking). No change needed.

Notes

  • The sandbox may not have bun or tsc available due to network restrictions. If so, the evaluation should install them or use a fallback. Check what's available in the autoloop runner environment.
  • An alternative to evaluation-time checks: add a pre-accept hook in the autoloop workflow that runs the type checker. But evaluation-time is simpler and already has the rejection mechanism.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions