DEV-1524: add claude_sdk_otf_raw + claude_sdk_otf_ainteract_raw agents

[ZmeiGorynych]

Summary

• Adds two new raw-SQL Claude SDK OTF agents that mirror the SLayer OTF variants but use no SLayer at all — agents issue raw SQL via the BIRD tool suite (execute_sql, get_schema, etc.) and submit via submit_sql
• ClaudeSDKOtfRawAgent — bound to --dataset livesqlbench --mode one-shot --query-mode raw; mirrors ClaudeSDKOtfAgent
• ClaudeSDKOtfAInteractRawAgent — bound to --dataset mini_interact --mode a-interact --query-mode raw; mirrors ClaudeSDKOtfAInteractAgent with the same ask_user-before-submit_sql gate (_make_ask_user_guards + nag hook)
• Refactors existing slayer OTF prompts (claude_sdk_otf/prompts.py, claude_sdk_otf_ainteract/prompts.py) to import shared constants from a new agents/_shared_otf_prompts.py; rendered prompts are verified byte-for-byte identical via SHA-256 snapshot tests
• Wires both frameworks into run.py (_FRAMEWORK_DATASET_MODE_BINDING, _validate_slayer_setup, _validate_one_shot_framework, _make_runner, --framework choices) and adds early returns in cloud/driver.py + cloud/ray_app.py (no SLayer artifacts to upload/download)

Test plan

• env -u SSH_AUTH_SOCK uv run --extra all --extra dev --extra pydantic-ai pytest — 2020 passed, 95 skipped
• tests/test_shared_otf_prompts.py — SHA-256 snapshot tests verify slayer prompts are unchanged after refactor; structural tests cover shared constant format-placeholder contracts
• tests/test_claude_sdk_otf_raw_agent.py — 29 tests covering tool selection, prompt building, validation, turn-budget hook, usage, error paths
• tests/test_claude_sdk_otf_ainteract_raw_agent.py — 43 tests covering the above plus all three _make_ask_user_guards hooks (gate, counter, nag), state isolation across factory calls, and ask_user call tracking in usage output
• tests/test_claude_sdk_otf_raw_run_wiring.py — 29 tests covering run.py validation logic for both new frameworks

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Multi-variant audit emissions for contradictory authoritative sources; per-instance variant tracking and primary marking.
  • Task and submission annotation artifacts and inline grader to persist per-attempt evaluations.
  • Raw SQL "raw" query-mode agents for one-shot and interactive runs; baked annotation support in images.

• Documentation

  • Audit contract and LiveSQLBench procedures updated with contradiction checks and multi-variant rules.
  • README: cloud smoke defaults and multi-instance guidance.

[linear]

DEV-1524

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds DEV-1515 annotation schemas/IO, a tolerant cascading grader, multi-variant audited-gold behavior, raw OTF agents and prompt fragments, cloud/local inline grading and aggregation (cascading_phase1), image bake-in for annotations, consolidation/convert scripts, and wide test coverage.

Changes

DEV-1515 End-to-End Overhaul

Layer / File(s)	Summary
Annotation schema and IO src/bird_interact_agents/eval/annotation_schema.py, src/bird_interact_agents/eval/annotation_io.py, src/bird_interact_agents/eval/implicit_annotation.py	Adds Pydantic schemas for TaskAnnotation/SubmissionAnnotation, path helpers, read/write with validation, and implicit in-memory annotation builder.
Tolerant grader and inline grader src/bird_interact_agents/eval/tolerant_grader.py, src/bird_interact_agents/eval/grade_in_place.py, src/bird_interact_agents/eval/cascading_report.py	Implements cascade N1–N9 comparators, LLM-judge + cache, grade_submission orchestration, miss diagnostics, and emit_cascading_eval_json aggregator.
Agents, prompts, and OTF raw adapters src/bird_interact_agents/agents/*, src/bird_interact_agents/agents/_shared_otf_prompts.py, src/bird_interact_agents/agents/claude_sdk_otf_raw/*, src/bird_interact_agents/agents/claude_sdk_otf_ainteract_raw/*	Adds shared prompt fragments, refactors SLAYER prompts, and introduces raw OTF one-shot and a-interact raw agents with ask-before-submit hooks and usage accounting.
Cloud/local wiring and image build src/bird_interact_agents/cloud/*, src/bird_interact_agents/run.py, src/bird_interact_agents/paths.py, src/bird_interact_agents/results_db.py, Dockerfile.cloud, src/bird_interact_agents/cloud/image.py	Adds submission_annotation upload/read, merge on fetch, inline grading in ray worker, cascading metrics emission, annotations_root, image build context and hashing for annotations, and DB schema adjustments (observe-only fields).
Audited-gold multi-variant and consolidation .claude/skills/*, src/bird_interact_agents/cloud/_audited_gold_check.py, src/bird_interact_agents/harness.py, scripts/consolidate_mini_interact_audited.py, scripts/dev1515_households_14_multivariant.py	Documents contradiction check and multi-variant emission semantics; implements primary-preference indexing, consolidation script, and example instance multi-variant patching script.
Scripts & tooling scripts/* (dev1515_*, consolidate_*, generate_annotation_skeletons.py)	Adds conversion, regrade, cascade-summary, remap/reclassify, merge, and skeleton-generation utilities for DEV-1515.
Tests and docs tests/*, .claude/skills/*, README.md, .gitignore	Extensive new and updated tests covering schemas, grader comparators, orchestration, cloud merge/wiring, agent behaviors, prompts, and legacy-field removal; docs updated for procedures and image bake-in.

Sequence Diagram(s)

sequenceDiagram
  participant CLI as Local/Cloud CLI
  participant Runner as Agent Runner
  participant Grader as grade_one_submission
  participant Store as Annotations/checkout
  participant GCS as Cloud Storage

  CLI->>Runner: run_evaluation / submit
  Runner->>Grader: grade_one_submission(instance)
  Grader-->>Runner: submission_annotation.json (cascade)
  Runner->>Store: write annotations/<benchmark>/<db>/<instance>.submission.<run>.json
  Runner->>Store: emit_cascading_eval_json(out_patch)
  Runner->>GCS: upload submission_annotation.json
  Runner->>GCS: upload attempt row/log

Loading

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

Possibly related PRs

• MotleyAI/bird-agents#16 — Related DEV-1515 grader/cascade and annotation pipeline work.
• MotleyAI/bird-agents#6 — Overlaps on submit/eval plumbing and diagnostic field changes.
• MotleyAI/bird-agents#14 — Prior audited-gold overlay/layout changes that this PR extends for single-file multi-variant handling.

Poem

A rabbit checks each variant twice,
Keeps audit rows neat, not once but thrice.
Cascades hum from N1 to nine,
Raw agents ask, then cross the line.
Hop—annotations baked and bright, the tests all pass tonight. 🐇✨

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch egor/dev-1524-slayer-less-claude-sdk-agent

⚔️ Resolve merge conflicts

• Resolve merge conflict in branch egor/dev-1524-slayer-less-claude-sdk-agent

[coderabbitai]

Actionable comments posted: 5

🧹 Nitpick comments (5)

src/bird_interact_agents/eval/annotate.py (1)

212-216: ⚡ Quick win

Consider error handling for missing attempt file.

The code reads attempt-1.json without handling potential file-not-found or JSON parsing errors. While the docstring mentions tests pass a stub grader, production usage could encounter missing or malformed attempt files.

🛡️ Proposed defensive handling

     attempt_path = Path(rows_dir) / instance_id / "attempt-1.json"
+    if not attempt_path.exists():
+        raise FileNotFoundError(
+            f"Missing attempt file for {instance_id}: {attempt_path}"
+        )
+    try:
-    attempt = json.loads(attempt_path.read_text())
+        attempt = json.loads(attempt_path.read_text())
+    except json.JSONDecodeError as e:
+        raise ValueError(
+            f"Invalid JSON in attempt file {attempt_path}: {e}"
+        ) from e
     submitted_sql = attempt.get("submitted_sql", "")

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/bird_interact_agents/eval/annotate.py` around lines 212 - 216, The code
assumes attempt-1.json always exists and is valid; update the logic around
attempt_path, attempt and the subsequent submitted_sql/traj/usage extraction to
handle missing or malformed files by first checking attempt_path.exists(), and
wrapping attempt_path.read_text() + json.loads(...) in a try/except that catches
FileNotFoundError and json.JSONDecodeError; on error either log/raise a clear
message including attempt_path and instance_id or set attempt = {} so
submitted_sql, traj and usage fall back to their existing defaults, ensuring the
rest of the function (where submitted_sql, traj, usage are used) won’t crash.

tests/test_claude_sdk_otf_ainteract_raw_agent.py (1)

257-258: ⚡ Quick win

Prefix unused unpacked variables with underscore.

The test unpacks counter_a and nag_b but never uses them. Prefix with underscore to signal the intentional omission.

✨ Suggested fix

-    gate_a, counter_a, nag_a = m._make_ask_user_guards()
-    gate_b, counter_b, nag_b = m._make_ask_user_guards()
+    gate_a, _counter_a, nag_a = m._make_ask_user_guards()
+    gate_b, counter_b, _nag_b = m._make_ask_user_guards()

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/test_claude_sdk_otf_ainteract_raw_agent.py` around lines 257 - 258, The
test unpacks values from m._make_ask_user_guards into gate_a, counter_a, nag_a
and gate_b, counter_b, nag_b but others are unused; update the unpacking to
prefix unused variables with underscores (e.g., keep gate_a and nag_a as-is if
used, change counter_a to _counter_a, and change counter_b or nag_b to
_counter_b/_nag_b as appropriate) so intent is clear and linters won't flag
unused variables.

src/bird_interact_agents/eval/tolerant_grader.py (1)

221-224: ⚡ Quick win

Consider adding strict=True to zip() for defensive coding.

While the length check at line 219 protects against mismatched row lengths, explicitly passing strict=True to zip() would make the invariant more explicit and catch unexpected length mismatches earlier in future refactorings.

🛡️ Proposed change

             if all(
-                _numeric_cell_equal(a, b, epsilon=epsilon)
-                for a, b in zip(pr, gr)
+                _numeric_cell_equal(a, b, epsilon=epsilon)
+                for a, b in zip(pr, gr, strict=True)
             ):

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/bird_interact_agents/eval/tolerant_grader.py` around lines 221 - 224, Add
strict=True to the zip call in the equality check to make the row-length
invariant explicit: inside the block using _numeric_cell_equal(...) for a, b in
zip(pr, gr) (with epsilon passed through), change the iterator to zip(pr, gr,
strict=True) so any unexpected length mismatch between pr and gr fails
immediately even if there is a prior length check. Ensure the import/target
runtime supports Python 3.10+ where zip(..., strict=True) is available.

src/bird_interact_agents/eval/cascading_report.py (1)

86-92: ⚡ Quick win

Prefer deriving delta keys from the counts dict.

The deltas loop hardcodes "n1" through "n9", while line 73 dynamically derives count keys from _CASCADE_ORDER. If _CASCADE_ORDER is extended or reordered, the deltas loop won't automatically adapt, creating a maintenance hazard.

♻️ Derive delta keys from counts

-    for k in ("n1", "n2", "n3", "n4", "n5", "n6", "n7", "n8", "n9"):
+    for k in counts.keys():
         if prev is None:
             deltas[k] = 0
         else:

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/bird_interact_agents/eval/cascading_report.py` around lines 86 - 92, The
delta computation currently hardcodes keys "n1".."n9"; instead iterate the keys
derived from the same source as counts (e.g., iterate over _CASCADE_ORDER or
list(counts.keys()) in the same order used to build counts) and compute deltas
into deltas[k] using the existing prev logic (set delta 0 for the first key,
else counts[k] - prev, then set prev = counts[k]); update references to prev,
deltas, counts and _CASCADE_ORDER so the delta loop adapts if the cascade order
changes.

src/bird_interact_agents/eval/regrade.py (1)

121-127: 💤 Low value

Consider making the gold sidecar filename configurable.

The hardcoded filename "livesqlbench_sqlite_gt_kg_testcases_0528.jsonl" at line 123 could become stale if the gold sidecar is updated. While the code safely checks candidate.exists() and respects the BIRD_LIVESQLBENCH_GOLD_FILE env override, a future-proof approach would source the filename from benchmark configuration or make it a CLI option.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/bird_interact_agents/eval/regrade.py` around lines 121 - 127, The code
currently hardcodes the sidecar filename
"livesqlbench_sqlite_gt_kg_testcases_0528.jsonl" when searching under
paths.benchmark_data_root(benchmark) — change this to use a configurable value
(preferably from the benchmark configuration or a CLI flag) and fall back to the
existing BIRD_LIVESQLBENCH_GOLD_FILE env override and then a sensible default;
update the lookup around gold_path and the loop that checks candidate.exists()
to reference that configurable name (e.g., use benchmark.get("gold_sidecar") or
a parsed CLI option) so future changes to the gold filename don’t require code
edits.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/consolidate_mini_interact_audited.py`:
- Around line 23-24: Replace the repo-root-based path computation (ROOT and
AUDITED) with the path helper: remove the Path(__file__)-derived ROOT and
AUDITED variables and instead call paths.audited_gold_root(); add an import for
the helper (from bird_interact_agents import paths) and update any references to
AUDITED to use the value returned by paths.audited_gold_root() so the script
honors git worktrees and the BIRD_AUDITED_GOLD_ROOT override.

In `@src/bird_interact_agents/eval/annotation_io.py`:
- Around line 47-59: The function _annotations_root currently constructs
Path(repo_root) / ANNOTATIONS_DIRNAME which bypasses the centralized helper;
change it to always resolve via the paths helper (call paths.annotations_root()
and do not build repo_root/ANNOTATIONS_DIRNAME directly). Update the
implementation of _annotations_root to return paths.annotations_root()
unconditionally (or, if you must keep the repo_root parameter, forward it to
paths.annotations_root(repo_root) rather than concatenating ANNOTATIONS_DIRNAME
yourself) so ANNOTATIONS_DIRNAME is resolved by the canonical
paths.annotations_root helper.
- Around line 61-73: The path builders (e.g., task_annotation_path and the
similar functions around lines 76-89) place user-provided components
(selected_database, instance_id, run_id) directly into filesystem paths and need
sanitization to prevent path traversal; update each function to validate and
normalize these components by ensuring each is a single path segment (no path
separators), not equal to '.' or '..', and using the basename form (e.g.,
Path(value).name) or explicit checks, and raise a ValueError on invalid input
before composing the final Path with _annotations_root and _canonical_benchmark
so no supplied value can escape the annotations tree.

In `@src/bird_interact_agents/run.py`:
- Around line 1129-1153: The inline grader reconstructs per-task DB path
(per_task_db) from data_dir/selected_database/... which ignores the materialized
local copy placed into task_data by materialize_task_db; replace the constructed
per_task_db with the materialized path from the task data (e.g.,
td.get("db_file_path") or task_data["db_file_path"]) when calling
grade_one_submission so the grader uses the same sqlite file the agent executed
against (ensure it's converted to a Path if needed and still passed as the
db_path argument to grade_one_submission).

In `@tests/test_local_run_cascading.py`:
- Around line 30-32: The FakeExecutor.__call__ currently returns identical
results for the conditional, so update it to return distinct outcomes for the
submitted SQL vs the other case: inside FakeExecutor.__call__ (method name
__call__, class FakeExecutor) keep the if sql == submitted branch returning the
intended "submitted" tuple and change the else branch to return a different
tuple representing the "gold"/original result (e.g., different rows or column
names) so the grader can distinguish submitted vs original execution results.

---

Nitpick comments:
In `@src/bird_interact_agents/eval/annotate.py`:
- Around line 212-216: The code assumes attempt-1.json always exists and is
valid; update the logic around attempt_path, attempt and the subsequent
submitted_sql/traj/usage extraction to handle missing or malformed files by
first checking attempt_path.exists(), and wrapping attempt_path.read_text() +
json.loads(...) in a try/except that catches FileNotFoundError and
json.JSONDecodeError; on error either log/raise a clear message including
attempt_path and instance_id or set attempt = {} so submitted_sql, traj and
usage fall back to their existing defaults, ensuring the rest of the function
(where submitted_sql, traj, usage are used) won’t crash.

In `@src/bird_interact_agents/eval/cascading_report.py`:
- Around line 86-92: The delta computation currently hardcodes keys "n1".."n9";
instead iterate the keys derived from the same source as counts (e.g., iterate
over _CASCADE_ORDER or list(counts.keys()) in the same order used to build
counts) and compute deltas into deltas[k] using the existing prev logic (set
delta 0 for the first key, else counts[k] - prev, then set prev = counts[k]);
update references to prev, deltas, counts and _CASCADE_ORDER so the delta loop
adapts if the cascade order changes.

In `@src/bird_interact_agents/eval/regrade.py`:
- Around line 121-127: The code currently hardcodes the sidecar filename
"livesqlbench_sqlite_gt_kg_testcases_0528.jsonl" when searching under
paths.benchmark_data_root(benchmark) — change this to use a configurable value
(preferably from the benchmark configuration or a CLI flag) and fall back to the
existing BIRD_LIVESQLBENCH_GOLD_FILE env override and then a sensible default;
update the lookup around gold_path and the loop that checks candidate.exists()
to reference that configurable name (e.g., use benchmark.get("gold_sidecar") or
a parsed CLI option) so future changes to the gold filename don’t require code
edits.

In `@src/bird_interact_agents/eval/tolerant_grader.py`:
- Around line 221-224: Add strict=True to the zip call in the equality check to
make the row-length invariant explicit: inside the block using
_numeric_cell_equal(...) for a, b in zip(pr, gr) (with epsilon passed through),
change the iterator to zip(pr, gr, strict=True) so any unexpected length
mismatch between pr and gr fails immediately even if there is a prior length
check. Ensure the import/target runtime supports Python 3.10+ where zip(...,
strict=True) is available.

In `@tests/test_claude_sdk_otf_ainteract_raw_agent.py`:
- Around line 257-258: The test unpacks values from m._make_ask_user_guards into
gate_a, counter_a, nag_a and gate_b, counter_b, nag_b but others are unused;
update the unpacking to prefix unused variables with underscores (e.g., keep
gate_a and nag_a as-is if used, change counter_a to _counter_a, and change
counter_b or nag_b to _counter_b/_nag_b as appropriate) so intent is clear and
linters won't flag unused variables.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 3d1d7544-8850-4903-9e69-1faf6978952b

📥 Commits

Reviewing files that changed from the base of the PR and between 81a6751 and d57d0a4.

📒 Files selected for processing (105)

• .claude/skills/_shared/audit-gold-sql.contract.md
• .claude/skills/annotate-task-submission/SKILL.md
• .claude/skills/audit-gold-sql-livesqlbench/SKILL.md
• .claude/skills/audit-gold-sql/SKILL.md
• .gitignore
• Dockerfile.cloud
• README.md
• analyses/households_failure_analysis_20260531t1008-claudes-slayer-890419.md
• analyses/museum_failure_analysis_20260531t1013-claudes-slayer-48eb0f.md
• analyses/raw/cross_cutting_observations.md
• analyses/raw/households_10.md
• analyses/raw/households_12.md
• analyses/raw/households_15.md
• analyses/raw/households_2.md
• analyses/raw/museum_10.md
• analyses/raw/museum_2.md
• analyses/raw/museum_3.md
• analyses/raw/museum_4.md
• analyses/raw/museum_5.md
• analyses/raw/museum_9.md
• scripts/consolidate_mini_interact_audited.py
• scripts/dev1515_cascade_summary.py
• scripts/dev1515_convert_livesqlbench.py
• scripts/dev1515_convert_runs.py
• scripts/dev1515_households_14_multivariant.py
• scripts/dev1515_reclassify_sufficiency.py
• scripts/dev1515_remap_failure_classes.py
• scripts/dev1515_strict_miss_diagnostics.py
• scripts/generate_annotation_skeletons.py
• scripts/verify_audited_gold.py
• src/bird_interact_agents/agents/_shared_otf_prompts.py
• src/bird_interact_agents/agents/_submit.py
• src/bird_interact_agents/agents/agno/agent.py
• src/bird_interact_agents/agents/claude_sdk/agent.py
• src/bird_interact_agents/agents/claude_sdk_otf/agent.py
• src/bird_interact_agents/agents/claude_sdk_otf/prompts.py
• src/bird_interact_agents/agents/claude_sdk_otf_ainteract/agent.py
• src/bird_interact_agents/agents/claude_sdk_otf_ainteract/prompts.py
• src/bird_interact_agents/agents/claude_sdk_otf_ainteract_raw/__init__.py
• src/bird_interact_agents/agents/claude_sdk_otf_ainteract_raw/agent.py
• src/bird_interact_agents/agents/claude_sdk_otf_ainteract_raw/prompts.py
• src/bird_interact_agents/agents/claude_sdk_otf_raw/__init__.py
• src/bird_interact_agents/agents/claude_sdk_otf_raw/agent.py
• src/bird_interact_agents/agents/claude_sdk_otf_raw/prompts.py
• src/bird_interact_agents/agents/mcp_agent/agent.py
• src/bird_interact_agents/agents/pydantic_ai/agent.py
• src/bird_interact_agents/agents/pydantic_ai_otf_encode/agent.py
• src/bird_interact_agents/agents/pydantic_ai_recursive/agent.py
• src/bird_interact_agents/agents/smolagents/agent.py
• src/bird_interact_agents/benchmark.py
• src/bird_interact_agents/cloud/_audited_gold_check.py
• src/bird_interact_agents/cloud/cli.py
• src/bird_interact_agents/cloud/collation.py
• src/bird_interact_agents/cloud/driver.py
• src/bird_interact_agents/cloud/gcs.py
• src/bird_interact_agents/cloud/image.py
• src/bird_interact_agents/cloud/post_run_merge.py
• src/bird_interact_agents/cloud/ray_app.py
• src/bird_interact_agents/eval/__init__.py
• src/bird_interact_agents/eval/annotate.py
• src/bird_interact_agents/eval/annotation_io.py
• src/bird_interact_agents/eval/annotation_schema.py
• src/bird_interact_agents/eval/cascading_report.py
• src/bird_interact_agents/eval/grade_in_place.py
• src/bird_interact_agents/eval/implicit_annotation.py
• src/bird_interact_agents/eval/regrade.py
• src/bird_interact_agents/eval/tolerant_grader.py
• src/bird_interact_agents/harness.py
• src/bird_interact_agents/paths.py
• src/bird_interact_agents/results_db.py
• src/bird_interact_agents/run.py
• tests/cloud/test_cli.py
• tests/cloud/test_collation.py
• tests/cloud/test_fetch_annotation_merge.py
• tests/cloud/test_fetch_cascading_phase1.py
• tests/cloud/test_image_annotations.py
• tests/cloud/test_inline_grader.py
• tests/test_benchmark.py
• tests/test_cascading_report.py
• tests/test_claude_sdk_otf_agent.py
• tests/test_claude_sdk_otf_ainteract_agent.py
• tests/test_claude_sdk_otf_ainteract_raw_agent.py
• tests/test_claude_sdk_otf_raw_agent.py
• tests/test_claude_sdk_otf_raw_run_wiring.py
• tests/test_claude_sdk_usage.py
• tests/test_dual_eval.py
• tests/test_eval_annotate_cli.py
• tests/test_eval_annotation_schema.py
• tests/test_implicit_task_annotation.py
• tests/test_legacy_field_removal.py
• tests/test_livesqlbench_audited_gold.py
• tests/test_local_run_cascading.py
• tests/test_miss_diagnostics.py
• tests/test_normalize_sol_sql.py
• tests/test_paths.py
• tests/test_paths_annotations.py
• tests/test_regrade_cli.py
• tests/test_results_db.py
• tests/test_run_local_inline_grader.py
• tests/test_schema_extension.py
• tests/test_shared_otf_prompts.py
• tests/test_tolerant_grader_comparators.py
• tests/test_tolerant_grader_multi_sql_conn.py
• tests/test_tolerant_grader_orchestration.py
• tests/test_verdict_label_shared.py

💤 Files with no reviewable changes (9)

• tests/test_claude_sdk_otf_agent.py
• src/bird_interact_agents/agents/agno/agent.py
• src/bird_interact_agents/agents/smolagents/agent.py
• src/bird_interact_agents/agents/mcp_agent/agent.py
• src/bird_interact_agents/agents/pydantic_ai/agent.py
• src/bird_interact_agents/agents/claude_sdk/agent.py
• src/bird_interact_agents/agents/pydantic_ai_otf_encode/agent.py
• src/bird_interact_agents/agents/pydantic_ai_recursive/agent.py
• tests/test_claude_sdk_usage.py

[coderabbitai]

🧹 Nitpick comments (1)

src/bird_interact_agents/eval/regrade.py (1)

194-221: ⚡ Quick win

Guard the attempt-JSON load so one corrupt file doesn't abort the whole re-grade.

json.loads(attempt.read_text()) on Line 198 sits outside the try/except that begins on Line 209. A single truncated or malformed attempt-N.json (e.g. an interrupted write) raises JSONDecodeError and aborts the entire offline regrade rather than skipping that one instance — the opposite of the resilient per-instance behavior the grader-exception handler already provides.

♻️ Skip unreadable attempt files instead of aborting

         attempt = _latest_attempt_file(sub)
         if attempt is None:
             report.skipped += 1
             continue
-        attempt_data = json.loads(attempt.read_text())
+        try:
+            attempt_data = json.loads(attempt.read_text())
+        except (json.JSONDecodeError, OSError) as exc:
+            print(f"  skip {instance_id}: unreadable attempt {attempt.name}: {exc}")
+            report.skipped += 1
+            continue
         submitted_sql = attempt_data.get("submitted_sql", "")

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/bird_interact_agents/eval/regrade.py` around lines 194 - 221, The JSON
load of per-attempt data (attempt_data = json.loads(attempt.read_text())) must
be guarded so a corrupted attempt-N.json doesn't abort the whole regrade: wrap
the file read + json.loads in the existing per-instance try/except (or add a
small try/except) around that operation, catch JSONDecodeError (or Exception)
and on error print a short message including instance_id and the exception,
increment report.skipped, and continue; keep the rest of the grader invocation
(grader(...), report.skipped handling) unchanged so only unreadable attempt
files are skipped.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/bird_interact_agents/eval/regrade.py`:
- Around line 194-221: The JSON load of per-attempt data (attempt_data =
json.loads(attempt.read_text())) must be guarded so a corrupted attempt-N.json
doesn't abort the whole regrade: wrap the file read + json.loads in the existing
per-instance try/except (or add a small try/except) around that operation, catch
JSONDecodeError (or Exception) and on error print a short message including
instance_id and the exception, increment report.skipped, and continue; keep the
rest of the grader invocation (grader(...), report.skipped handling) unchanged
so only unreadable attempt files are skipped.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 47c6b682-af6a-446b-8c0f-ca10e4865803

📥 Commits

Reviewing files that changed from the base of the PR and between d57d0a4 and 6b59ba1.

📒 Files selected for processing (17)

• scripts/consolidate_mini_interact_audited.py
• scripts/dev1515_convert_runs.py
• src/bird_interact_agents/cloud/ray_app.py
• src/bird_interact_agents/eval/annotate.py
• src/bird_interact_agents/eval/annotation_schema.py
• src/bird_interact_agents/eval/cascading_report.py
• src/bird_interact_agents/eval/regrade.py
• src/bird_interact_agents/eval/tolerant_grader.py
• src/bird_interact_agents/run.py
• tests/test_cascading_report.py
• tests/test_eval_annotation_schema.py
• tests/test_local_run_cascading.py
• tests/test_regrade_cli.py
• tests/test_run_local_inline_grader.py
• tests/test_tolerant_grader_comparators.py
• tests/test_tolerant_grader_orchestration.py
• tests/test_user_sim_interaction_trajectory.py

🚧 Files skipped from review as they are similar to previous changes (11)

• scripts/consolidate_mini_interact_audited.py
• tests/test_local_run_cascading.py
• src/bird_interact_agents/cloud/ray_app.py
• scripts/dev1515_convert_runs.py
• src/bird_interact_agents/eval/annotate.py
• src/bird_interact_agents/eval/annotation_schema.py
• tests/test_cascading_report.py
• src/bird_interact_agents/eval/cascading_report.py
• tests/test_run_local_inline_grader.py
• src/bird_interact_agents/run.py
• src/bird_interact_agents/eval/tolerant_grader.py