DEV-1525: clean up handling of multiple benchmarks

[ZmeiGorynych]

Summary

• Benchmark renames: All 4 benchmarks use official hyphenated names (mini-interact, livesqlbench-base-lite-sqlite, livesqlbench-base-lite, bird-interact-lite-exp). Old underscore tokens are rejected with a clear error.
• Uniform OTF paths: slayer_otf_cache/<benchmark>/ and slayer_models_otf/<benchmark>/ — a single env var per helper controls the parent dir; benchmark name is the subdir.
• Gated gold: New gated_gold_root(*, benchmark) path helper replaces the removed Benchmark.gold_root_env field.
• CLI unification: --framework claude_sdk is the only public option; _make_runner dispatches to the right internal OTF agent based on (benchmark.one_shot, query_mode). Removed _validate_framework_dataset_mode, _validate_one_shot_framework, _FRAMEWORK_DATASET_MODE_BINDING.
• No silent defaults: All or "mini_interact" / or "livesqlbench" fallbacks removed — hard ValueError raises instead. ray_app.py --dataset is now required.
• Removed shims: mini_interact_root(), mini_interact_data_file(), livesqlbench_root(), livesqlbench_data_file() deleted; callers updated to benchmark_data_root("mini-interact") etc.
• Migration script: scripts/migrate_benchmark_paths.py renames existing artifact dirs (OTF cache/models, audited gold, annotations) to match the new layout.

Test plan

• Full non-integration test suite: 2350 passed, 0 failed
• uv run python -c "from bird_interact_agents import paths; print(paths.slayer_otf_cache_root(benchmark='mini-interact'))" → …/slayer_otf_cache/mini-interact
• uv run python -c "from bird_interact_agents.benchmark import get_benchmark; print(get_benchmark('mini-interact').name)" → mini-interact
• Old tokens rejected: get_benchmark('mini_interact') → ValueError: unknown benchmark token
• Migration script dry-run is a no-op on clean worktree

🤖 Generated with Claude Code

Summary by CodeRabbit

• Documentation

  • README updated: canonical hyphenated benchmark names, rewritten Quick Start examples, Annotator Agent section (use annotate), and updated one-shot/SLayer guidance.

• Chores

  • Standardized per‑benchmark data roots and paths; LiveSQLBench targets livesqlbench-base-lite-sqlite.

• User-facing changes

  • CLI: framework restricted to claude_sdk; dataset must be provided (hyphenated tokens); slayer requires --slayer-setup on-the-fly.

• Cloud

  • Gated‑gold files are included in dataset uploads/downloads and accepted as gold-file locations.

[linear]

DEV-1525

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: a8e2dcf9-9a47-48ea-b2ee-e0a48b97b914

📥 Commits

Reviewing files that changed from the base of the PR and between 160bff1 and 7ccd6e4.

📒 Files selected for processing (8)

• README.md
• src/bird_interact_agents/agents/annotator/agent.py
• src/bird_interact_agents/cloud/driver.py
• src/bird_interact_agents/eval/annotation_schema.py
• src/bird_interact_agents/eval/regrade.py
• tests/test_annotator_tools.py
• tests/test_eval_annotation_schema.py
• tests/test_paths.py

✅ Files skipped from review due to trivial changes (1)

• README.md

🚧 Files skipped from review as they are similar to previous changes (6)

• src/bird_interact_agents/eval/annotation_schema.py
• src/bird_interact_agents/cloud/driver.py
• tests/test_eval_annotation_schema.py
• src/bird_interact_agents/eval/regrade.py
• tests/test_paths.py
• tests/test_annotator_tools.py

---

📝 Walkthrough

Walkthrough

Canonicalize benchmark identifiers to hyphenated forms, require explicit dataset threading, add per-benchmark paths including gated-gold, restrict CLI/framework to claude_sdk, thread dataset through run/cloud flows, strengthen annotator validation/prompts/tools, and update scripts/tests to match.

Changes

Canonical benchmark naming and claude_sdk-only execution

Layer / File(s)	Summary
Benchmark registry and path helpers src/bird_interact_agents/benchmark.py, src/bird_interact_agents/paths.py, tests/test_paths*	Benchmarks use hyphenated canonical names; removed legacy shim helpers; introduced per-benchmark nested OTF/cache/model roots and gated_gold_root.
Run/CLI and cloud dispatch src/bird_interact_agents/run.py, src/bird_interact_agents/cloud/ray_app.py, src/bird_interact_agents/cloud/cli.py	--dataset is required, --framework restricted to claude_sdk; dataset threaded into runner creation and actor dispatch; _validate_framework_mode and simplified _validate_slayer_setup added.
Cloud driver and gated-gold handling src/bird_interact_agents/cloud/driver.py, src/bird_interact_agents/cloud/benchmark_data.py, src/bird_interact_agents/cloud/ray_app_annotator.py	Gold-file validation now allows benchmark data root or gated-gold root; ensure_uploaded() includes gated gold in content hash and uploads gated-gold subprefix.
Agent dataset gating src/bird_interact_agents/agents/*_agent.py, src/bird_interact_agents/run.py	Agents require task_data["dataset"] and validate get_benchmark(dataset).one_shot where applicable; run._make_runner dispatches Claude SDK variants by benchmark shape and query_mode.
Annotator validation & prompts src/bird_interact_agents/agents/annotator/agent.py, src/bird_interact_agents/agents/annotator/prompts.py, src/bird_interact_agents/eval/annotation_schema.py	TaskAnnotation validation expanded (evidence_sources prefix, audited-variant/gold_variant cross-checks, evaluator_prompt rules); deterministic provenance filling and get_column_sample_values tool added; prompts reorganized.
Annotation I/O and regrade src/bird_interact_agents/eval/annotation_io.py, src/bird_interact_agents/eval/regrade.py, src/bird_interact_agents/cloud/post_run_merge.py	Stop dash→underscore normalization; use canonical benchmark for annotation paths and gated-gold discovery.
Scripts & migration scripts/*, scripts/migrate_benchmark_paths.py	Updated default dataset roots to benchmark_data_root(...) and added one-off migration script to move artifacts to new benchmark-scoped layout.
Tests & README tests/... (many files), README.md	Massive test updates to use hyphenated benchmark tokens, thread dataset param, and assert new path/gated-gold behaviors; README updated for claude_sdk examples and annotator docs.

Sequence Diagram(s)

(Skipped — changes are broad plumbing and validation updates.)

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• MotleyAI/bird-agents#18 — both touch annotator agent implementation and prompts/validation.
• MotleyAI/bird-agents#10 — overlaps on agent dataset gating and one-shot validation logic.
• MotleyAI/bird-agents#9 — related changes to run/dispatch framework-dataset binding and runner plumbing.

Poem

A rabbit renames trails with tidy dashes,
Packs gold behind a gated stash that flashes.
One SDK to run the show,
Datasets declared so flows can go.
Paths align, tests hop—hip-hop! 🐇✨

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch egor/dev-1525-clean-up-handling-of-multiple-benchmarks

[coderabbitai]

Actionable comments posted: 13

🧹 Nitpick comments (4)

tests/test_paths.py (1)

527-527: 💤 Low value

Optional: Consider prefixing unused variables with underscore.

Static analysis flags that the unpacked main variable is unused in these test setup calls. While not a functional issue, prefixing with _main would silence the warning.

Suggested change

For line 527:

-    main, _wt = _setup_main_and_worktree(tmp_path, monkeypatch)
+    _main, _wt = _setup_main_and_worktree(tmp_path, monkeypatch)

For line 584:

-    main, _wt = _setup_main_and_worktree(tmp_path, monkeypatch)
+    _main, _wt = _setup_main_and_worktree(tmp_path, monkeypatch)

Also applies to: 584-584

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/test_paths.py` at line 527, The test unpacks main and _wt from
_setup_main_and_worktree but static analysis reports main is unused; rename the
variable to _main (or use _ if preferred) in the unpack at the call to
_setup_main_and_worktree to silence the warning, and make the same change for
the other occurrence referenced (the similar unpack at the second call around
line 584) so both unused unpacked values are prefixed with an underscore.

src/bird_interact_agents/cloud/post_run_merge.py (1)

600-601: 💤 Low value

Consider removing the now-redundant _normalise_benchmark helper.

Since the function now returns its input unchanged (no normalization), it serves no purpose and could be removed. Direct usage of the benchmark parameter in merge_task_annotations() (line 616) and merge_audited_gold_variants() (line 662) would simplify the code without changing behavior.

♻️ Proposed simplification

Remove the function:

-def _normalise_benchmark(benchmark: str) -> str:
-    return benchmark
-
-

And replace its two call sites:

 def merge_task_annotations(
     *,
     downloaded_run_dir: Path,
     benchmark: str,
     annotations_root: Path,
 ) -> TaskAnnotationMergeReport:
-    bm = _normalise_benchmark(benchmark)
+    bm = benchmark
     report = TaskAnnotationMergeReport()

 def merge_audited_gold_variants(
     *,
     downloaded_run_dir: Path,
     benchmark: str,
     audited_gold_root: Path,
     override: bool = False,
 ) -> AuditedGoldVariantsMergeReport:
-    bm = _normalise_benchmark(benchmark)
+    bm = benchmark
     report = AuditedGoldVariantsMergeReport()

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/bird_interact_agents/cloud/post_run_merge.py` around lines 600 - 601, The
helper _normalise_benchmark is redundant because it returns its input unchanged;
remove the _normalise_benchmark function and replace its call sites in
merge_task_annotations and merge_audited_gold_variants to pass the benchmark
parameter directly (i.e., use the benchmark variable instead of calling
_normalise_benchmark(benchmark)); ensure any imports or tests referencing
_normalise_benchmark are updated/removed and run tests to verify no behavior
change.

tests/test_claude_sdk_otf_ainteract_run_wiring.py (1)

208-208: 💤 Low value

Consider using ASCII x instead of × in docstring.

Static analysis flags the multiplication sign character in the docstring. For consistency and to avoid encoding issues, consider using a regular ASCII x or the word "and".

📝 Optional style fix

-    """Dataset×mode gate: livesqlbench doesn't support a-interact."""
+    """Dataset x mode gate: livesqlbench doesn't support a-interact."""

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/test_claude_sdk_otf_ainteract_run_wiring.py` at line 208, The
module/test docstring contains a multiplication sign '×' which static analysis
flags; update the string in tests/test_claude_sdk_otf_ainteract_run_wiring.py
(the docstring around the dataset×mode gate) to use an ASCII 'x' or the word
"and" (e.g., "Dataset x mode gate: livesqlbench doesn't support a-interact.") to
avoid encoding issues and ensure consistency.

src/bird_interact_agents/agents/annotator/agent.py (1)

454-503: 💤 Low value

Consider using iterable unpacking for list extension.

At line 494-501, the code concatenates lists using +. This could be simplified using tuple unpacking in the list constructor.

♻️ Suggested refactoring

-                updated.masked_terms = list(updated.masked_terms) + [
-                    MaskedTerm(
-                        term=term,
-                        type=item.get("type", "knowledge_linking_ambiguity"),
-                        is_mask=True,
-                        metadata_evidence=evidence,
-                    )
-                ]
+                updated.masked_terms = [
+                    *updated.masked_terms,
+                    MaskedTerm(
+                        term=term,
+                        type=item.get("type", "knowledge_linking_ambiguity"),
+                        is_mask=True,
+                        metadata_evidence=evidence,
+                    ),
+                ]

Based on learnings: Static analysis tools flagged this pattern (RUF005).

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/bird_interact_agents/agents/annotator/agent.py` around lines 454 - 503,
In _fill_deterministic_fields, avoid concatenating lists with + when appending
the new MaskedTerm to updated.masked_terms; instead construct the new list using
iterable unpacking or .extend to preserve type and avoid RUF005 static-analysis
warnings (e.g., replace updated.masked_terms = list(updated.masked_terms) +
[MaskedTerm(...)] with updated.masked_terms = [*updated.masked_terms,
MaskedTerm(...)] or call updated.masked_terms.extend([MaskedTerm(...)]) inside
the function where MaskedTerm is created).

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/bird_interact_agents/cloud/benchmark_data.py`:
- Around line 120-125: The gated-gold directory upload currently calls
gcs.upload_dir_prefix(gated_gold, f"{prefix.rstrip('/')}/{GATED_GOLD_SUBDIR}",
client=client) without excluding VCS metadata, causing .git and similar dirs to
be uploaded even though _hash_dir_into() ignores them; update the call to pass
exclude=_is_vcs_path (or the existing VCS filter function used by
_hash_dir_into) so gcs.upload_dir_prefix(..., exclude=_is_vcs_path,
client=client) omits VCS files and keeps the uploaded file set consistent with
the hash.

In `@src/bird_interact_agents/cloud/upload_back.py`:
- Around line 240-243: The current code silently returns when dataset is missing
(cfg.get("dataset")), which hides misconfigured actors; change this to surface
the error by raising an exception (e.g., ValueError or RuntimeError) with a
clear message that includes the cfg or relevant keys so the missing `dataset`
invariant fails fast instead of quietly skipping. Locate the block that calls
cfg.get("dataset") and currently returns, and replace the silent return with a
raise that mentions the missing dataset and includes context (for example
referencing `cfg` and the attempted `get_benchmark(dataset)` usage) so callers
and the outer except can log/handle it properly.

In `@src/bird_interact_agents/eval/annotation_io.py`:
- Around line 40-43: Update the module-level docstring in
src/bird_interact_agents/eval/annotation_io.py to reflect the current behavior:
canonical benchmark names are hyphenated (e.g. "mini-interact"), no
normalization is applied (benchmark names are returned as-is), and there is no
shared dash/underscore tree or underscore-to-dash normalization; remove the
outdated mention of underscore normalization and shared trees and optionally
reference DEV-1525 for provenance.

In `@tests/cloud/test_annotator_gcs.py`:
- Around line 71-74: The test currently compares identical calls so it no longer
validates normalization; update the test for stable_task_annotation_blob to
assert the intended behavior: either replace the assertion with a context that
expects a ValueError when using an underscore token (call
gcs.stable_task_annotation_blob("mini_interact", "shop", "shop_1") and assert it
raises ValueError), or remove the test entirely if underscore/dash normalization
is no longer supported; locate the test function
test_stable_task_annotation_blob_normalises_dash_benchmark and change its body
to perform the appropriate raise-or-remove change for
gcs.stable_task_annotation_blob.
- Around line 83-86: The test currently compares two identical calls and no
longer verifies normalization; update
test_stable_audited_gold_variants_blob_normalises_dash_benchmark to assert that
gcs.stable_audited_gold_variants_blob rejects underscores: call
gcs.stable_audited_gold_variants_blob with a repo name containing an underscore
(e.g., "mini_interact") and wrap it in a pytest.raises(ValueError) to confirm it
raises, referencing the function name stable_audited_gold_variants_blob from
bird_interact_agents.cloud.gcs; alternatively, if you prefer removal, delete the
redundant test entirely.

In `@tests/cloud/test_annotator_merge.py`:
- Around line 217-223: Tests in the override-path cases still use the old
canonical name "mini_interact" and file "mini_interact_audited.jsonl"; update
those references to the new canonical names used earlier: change any
benchmark="mini_interact" to benchmark="mini-interact" and any filenames like
"mini_interact_audited.jsonl" to "mini-interact_audited.jsonl" (look for
occurrences in test functions and variables in
tests/cloud/test_annotator_merge.py such as the override-path tests that
construct downloaded_run_dir/audited_gold_root and consolidate into
mini_interact_audited.jsonl).

In `@tests/cloud/test_driver.py`:
- Around line 1766-1769: The assertion is too loose because 'or "/data/" in
result' allows unrelated in-cluster paths; update the test that calls
driver._in_cluster_gold_file(ns) to assert the result points exactly into the
benchmark's container directory by checking for the benchmark-specific path plus
the gold filename. Instruct the test to compute the expected fragment using the
benchmark name ("livesqlbench-base-lite-sqlite") and the basename of
ns.gold_file (or gold.name) and assert that expected_fragment in result (or that
result.startswith(f"/livesqlbench-base-lite-sqlite/{basename}")), removing the
generic '"/data/"' alternative so the assertion is precise.

In `@tests/test_annotator_agent.py`:
- Around line 643-658: The test uses a non-canonical benchmark id; update the
benchmark argument in test_fill_deterministic_fields_does_not_mutate_original to
use the hyphenated canonical name "mini-interact" instead of "mini_interact"
when calling _fill_deterministic_fields (reference the test function name and
the _fill_deterministic_fields call to locate the change).
- Around line 564-641: Replace non-canonical benchmark identifiers passed to
_fill_deterministic_fields: in
test_fill_deterministic_fields_no_duplicate_masked_terms change
benchmark="mini_interact" to benchmark="mini-interact", and in
test_fill_deterministic_fields_livesqlbench_skips_masked_terms change
benchmark="livesqlbench" to the canonical hyphenated identifier (e.g.,
"live-sql-bench"); update those string literals where _fill_deterministic_fields
is invoked so all benchmark names use hyphens consistently.

In `@tests/test_livesqlbench_loader.py`:
- Around line 122-125: The inline comment accompanying the assertion that checks
dataset values is stale; update the comment text to reflect the new canonical
dataset name used in the assertion. Locate the assertion that reads assert
all(r.get("dataset") == "livesqlbench-base-lite-sqlite" for r in rows) and
change the trailing comment which currently references
task['dataset']='livesqlbench' so it instead refers to
task['dataset']='livesqlbench-base-lite-sqlite' (or equivalent wording) to match
the actual checked value.

In `@tests/test_one_shot_mode.py`:
- Line 159: The guard test's argv for the one-shot/livesqlbench branch omits the
now-required --dataset flag so argparse fails before the
must_contain=["one-shot","livesqlbench"] assertion is reached; update the argv
array used in the test (the argv tuple/list near the "--framework", "claude_sdk"
entry in tests/test_one_shot_mode.py) to include an explicit non-livesqlbench
dataset, e.g. add "--dataset", "mini-interact", so the parser succeeds and the
one-shot/livesqlbench guard assertion is exercised.

In `@tests/test_pydantic_ai_otf_encode_run_wiring.py`:
- Around line 50-52: The test suite was partially migrated to the public
framework path but still calls run_evaluation(...
framework="pydantic_ai_otf_encode") and monkeypatches PydanticAIOtfEncodeAgent,
causing branch-selection mismatch; update the tests (the calls to run_evaluation
and any monkeypatch targets) to use the public framework string "claude_sdk" and
the corresponding agent/class path used for that public implementation so the
branch-selection assertions (e.g., choices = _framework_choices_from_parser())
and the constructor-path tests remain consistent with the rest of the file.

In `@tests/test_run_local_inline_grader.py`:
- Around line 196-199: Update the stale comment above the assertion that checks
benchmark values: change the comment that currently reads about the "canonical
underscore form" to reflect the actual expected hyphenated form
("mini-interact"). Locate the assertion referencing calls and the string
"mini-interact" (the line containing assert all(c["benchmark"] ==
"mini-interact" for c in calls)) and modify the preceding comment to accurately
state that the benchmark token is expected in hyphenated form.

---

Nitpick comments:
In `@src/bird_interact_agents/agents/annotator/agent.py`:
- Around line 454-503: In _fill_deterministic_fields, avoid concatenating lists
with + when appending the new MaskedTerm to updated.masked_terms; instead
construct the new list using iterable unpacking or .extend to preserve type and
avoid RUF005 static-analysis warnings (e.g., replace updated.masked_terms =
list(updated.masked_terms) + [MaskedTerm(...)] with updated.masked_terms =
[*updated.masked_terms, MaskedTerm(...)] or call
updated.masked_terms.extend([MaskedTerm(...)]) inside the function where
MaskedTerm is created).

In `@src/bird_interact_agents/cloud/post_run_merge.py`:
- Around line 600-601: The helper _normalise_benchmark is redundant because it
returns its input unchanged; remove the _normalise_benchmark function and
replace its call sites in merge_task_annotations and merge_audited_gold_variants
to pass the benchmark parameter directly (i.e., use the benchmark variable
instead of calling _normalise_benchmark(benchmark)); ensure any imports or tests
referencing _normalise_benchmark are updated/removed and run tests to verify no
behavior change.

In `@tests/test_claude_sdk_otf_ainteract_run_wiring.py`:
- Line 208: The module/test docstring contains a multiplication sign '×' which
static analysis flags; update the string in
tests/test_claude_sdk_otf_ainteract_run_wiring.py (the docstring around the
dataset×mode gate) to use an ASCII 'x' or the word "and" (e.g., "Dataset x mode
gate: livesqlbench doesn't support a-interact.") to avoid encoding issues and
ensure consistency.

In `@tests/test_paths.py`:
- Line 527: The test unpacks main and _wt from _setup_main_and_worktree but
static analysis reports main is unused; rename the variable to _main (or use _
if preferred) in the unpack at the call to _setup_main_and_worktree to silence
the warning, and make the same change for the other occurrence referenced (the
similar unpack at the second call around line 584) so both unused unpacked
values are prefixed with an underscore.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: e5cf50ff-2007-4349-9ca1-50cd1bd4d603

📥 Commits

Reviewing files that changed from the base of the PR and between bcecdd1 and 026fcc1.

📒 Files selected for processing (115)

• README.md
• scripts/append_deferred_stubs.py
• scripts/build_w4d_instructions.py
• scripts/compare_sar_vs_inhouse.py
• scripts/dev1515_households_14_multivariant.py
• scripts/export_slayer_models.py
• scripts/kb_op_audit_classify.py
• scripts/kb_op_audit_extract.py
• scripts/migrate_benchmark_paths.py
• scripts/multi_kb_audit.py
• scripts/prepare_livesqlbench.py
• scripts/redirect_kb_duplicates.py
• scripts/refresh_kb_annotations.py
• scripts/run_haiku_slayer.sh
• scripts/run_three_way.sh
• scripts/verify_audited_gold.py
• scripts/verify_kb_coverage.py
• src/bird_interact_agents/agents/annotator/agent.py
• src/bird_interact_agents/agents/annotator/prompts.py
• src/bird_interact_agents/agents/claude_sdk_otf/agent.py
• src/bird_interact_agents/agents/claude_sdk_otf_ainteract/agent.py
• src/bird_interact_agents/agents/claude_sdk_otf_ainteract_raw/agent.py
• src/bird_interact_agents/agents/claude_sdk_otf_raw/agent.py
• src/bird_interact_agents/agents/pydantic_ai_otf_encode/agent.py
• src/bird_interact_agents/agents/pydantic_ai_recursive/agent.py
• src/bird_interact_agents/benchmark.py
• src/bird_interact_agents/cloud/_audited_gold_check.py
• src/bird_interact_agents/cloud/benchmark_data.py
• src/bird_interact_agents/cloud/cli.py
• src/bird_interact_agents/cloud/driver.py
• src/bird_interact_agents/cloud/post_run_merge.py
• src/bird_interact_agents/cloud/ray_app.py
• src/bird_interact_agents/cloud/ray_app_annotator.py
• src/bird_interact_agents/cloud/upload_back.py
• src/bird_interact_agents/config.py
• src/bird_interact_agents/eval/annotate.py
• src/bird_interact_agents/eval/annotation_io.py
• src/bird_interact_agents/eval/grade_in_place.py
• src/bird_interact_agents/eval/implicit_annotation.py
• src/bird_interact_agents/eval/regrade.py
• src/bird_interact_agents/harness.py
• src/bird_interact_agents/paths.py
• src/bird_interact_agents/run.py
• src/bird_interact_agents/sar_audit/driver.py
• src/bird_interact_agents/slayer_pipeline/orchestrator.py
• tests/cloud/test_annotator_gcs.py
• tests/cloud/test_annotator_merge.py
• tests/cloud/test_annotator_ray_app.py
• tests/cloud/test_audited_gold_check.py
• tests/cloud/test_benchmark_data.py
• tests/cloud/test_cli.py
• tests/cloud/test_driver.py
• tests/cloud/test_fetch_annotation_merge.py
• tests/cloud/test_inline_grader.py
• tests/cloud/test_ray_app.py
• tests/cloud/test_run_one_task.py
• tests/cloud/test_upload_back.py
• tests/conftest.py
• tests/integration/test_livesqlbench_audited_gold_execute.py
• tests/sar_audit/test_cli_exit_code.py
• tests/sar_audit/test_verifier.py
• tests/test_annotator_agent.py
• tests/test_annotator_tools.py
• tests/test_audited_gold_overlay_wiring.py
• tests/test_autopsy.py
• tests/test_benchmark.py
• tests/test_benchmark_naming.py
• tests/test_benchmark_postgres.py
• tests/test_claude_sdk_otf_agent.py
• tests/test_claude_sdk_otf_ainteract_agent.py
• tests/test_claude_sdk_otf_ainteract_raw_agent.py
• tests/test_claude_sdk_otf_ainteract_run_wiring.py
• tests/test_claude_sdk_otf_raw_agent.py
• tests/test_claude_sdk_otf_raw_run_wiring.py
• tests/test_claude_sdk_otf_run_wiring.py
• tests/test_cloud_paths_unchanged.py
• tests/test_cloud_paths_uniform.py
• tests/test_db_connection.py
• tests/test_db_isolation.py
• tests/test_db_isolation_pre_copy.py
• tests/test_dual_eval.py
• tests/test_eval_annotate_cli.py
• tests/test_eval_annotation_schema.py
• tests/test_harness_postgres_dispatch.py
• tests/test_kb_memory_encoder.py
• tests/test_livesqlbench_audited_gold.py
• tests/test_livesqlbench_loader.py
• tests/test_miss_diagnostics.py
• tests/test_one_shot_mode.py
• tests/test_one_shot_otf_encode_factories.py
• tests/test_one_shot_recursive_factories.py
• tests/test_one_shot_run.py
• tests/test_otf_encode_reference_root.py
• tests/test_otf_postgres.py
• tests/test_otf_rebuild_per_benchmark.py
• tests/test_otf_rebuild_wiring.py
• tests/test_paths.py
• tests/test_paths_annotations.py
• tests/test_paths_uniform_otf.py
• tests/test_prepare_livesqlbench.py
• tests/test_projection_resolver.py
• tests/test_pydantic_ai_otf_encode_agent.py
• tests/test_pydantic_ai_otf_encode_run_wiring.py
• tests/test_pydantic_ai_recursive_trajectory.py
• tests/test_pydantic_ai_recursive_usage.py
• tests/test_recursive_runtime_db_root.py
• tests/test_regrade_cli.py
• tests/test_run_framework_dispatch.py
• tests/test_run_local_inline_grader.py
• tests/test_slayer_otf_smoke.py
• tests/test_slayer_setup_flag.py
• tests/test_submit_postgres.py
• tests/test_tolerant_grader_multi_sql_conn.py
• tests/test_tolerant_grader_orchestration.py
• tests/test_tolerant_grader_postgres.py

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@README.md`:
- Line 180: Replace underscore benchmark names with canonical hyphenated names:
change occurrences of "mini_interact" to "mini-interact" in the README examples
(specifically the CLI flag "--benchmark mini_interact") and any audited-gold
filename references so the CLI and file paths match the canonical benchmark
naming and won't be rejected.

In `@src/bird_interact_agents/agents/annotator/agent.py`:
- Around line 157-163: The prefill logic assumes ta_dict is a dict and its
"provenance" field is a dict which can raise on malformed JSON; guard by
checking that ta_dict (from task_annotation_json) is a mapping before calling
setdefault and if it's not, replace or coerce it to an empty dict, then ensure
provenance is a dict (if ta_dict.get("provenance") is not a dict, assign an
empty dict) before setting task_jsonl_path and task_jsonl_instance_id; update
the block around ta_dict, _prov, _benchmark_task_jsonl_name and task_data to
perform these type checks and safe assignments so TaskAnnotation.model_validate
runs and returns validation errors rather than raising.

In `@tests/test_annotator_tools.py`:
- Around line 1160-1168: The new test fixtures (e.g. variable
variant_int_primary and the other fixture blocks around the same change) use
benchmark: "mini_interact" but _setup_ctx now expects "mini-interact", causing
spurious benchmark-mismatch failures; update the benchmark token in those JSON
fixtures from "mini_interact" to "mini-interact" so the tests exercise the
intended primary-typing/duplicate-variant_id logic rather than failing early on
benchmark mismatch.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 372dd501-c7c5-4f81-8315-8b34a0615d86

📥 Commits

Reviewing files that changed from the base of the PR and between 026fcc1 and 160bff1.

📒 Files selected for processing (18)

• README.md
• src/bird_interact_agents/agents/annotator/agent.py
• src/bird_interact_agents/agents/annotator/prompts.py
• src/bird_interact_agents/cloud/benchmark_data.py
• src/bird_interact_agents/cloud/ray_app_annotator.py
• src/bird_interact_agents/cloud/upload_back.py
• src/bird_interact_agents/eval/annotation_io.py
• src/bird_interact_agents/eval/annotation_schema.py
• tests/cloud/test_annotator_gcs.py
• tests/cloud/test_annotator_merge.py
• tests/cloud/test_driver.py
• tests/test_annotator_agent.py
• tests/test_annotator_tools.py
• tests/test_eval_annotation_schema.py
• tests/test_livesqlbench_loader.py
• tests/test_one_shot_mode.py
• tests/test_pydantic_ai_otf_encode_run_wiring.py
• tests/test_run_local_inline_grader.py

✅ Files skipped from review due to trivial changes (2)

• tests/test_run_local_inline_grader.py
• tests/test_livesqlbench_loader.py

🚧 Files skipped from review as they are similar to previous changes (6)

• src/bird_interact_agents/agents/annotator/prompts.py
• src/bird_interact_agents/eval/annotation_io.py
• tests/test_annotator_agent.py
• tests/cloud/test_driver.py
• tests/test_one_shot_mode.py
• src/bird_interact_agents/cloud/upload_back.py