DEV-1525: clean up handling of multiple benchmarks - #26

Merged

ZmeiGorynych merged 26 commits into

mainfrom

egor/dev-1525-clean-up-handling-of-multiple-benchmarks

Jun 4, 2026

Commits on Jun 3, 2026

Merge remote-tracking branch 'origin/egor/dev-1518-annotator-agent' into egor/dev-1525-clean-up-handling-of-multiple-benchmarks
ZmeiGorynych
committed
6ad2599
annotator: fix annotation quality gaps found in smoke test

ZmeiGorynych
and
claude
committed
ced451e
DEV-1525: WIP — clean up handling of multiple benchmarks

ZmeiGorynych
and
claude
committed
d255008
Merge remote-tracking branch 'origin/main' into egor/dev-1525-clean-up-handling-of-multiple-benchmarks
ZmeiGorynych
committed
f0581e1
Merge remote-tracking branch 'origin/egor/dev-1518-annotator-agent' into egor/dev-1525-clean-up-handling-of-multiple-benchmarks
ZmeiGorynych
committed
5c315e6
annotator: fix metadata_evidence string→list and primary boolean validation

ZmeiGorynych
and
claude
committed
f308567
annotator: fix masked-term dedup to only block on is_mask=True entries

ZmeiGorynych
and
claude
committed
149d69a
DEV-1525: clean up handling of multiple benchmarks

ZmeiGorynych
and
claude
committed
611a92c
DEV-1525: fix migration script self-move bug; update docs/scripts with new benchmark and framework names

ZmeiGorynych
and
claude
committed
5d41293
annotator: validate solvable tasks require gold variants; insufficient tasks require evaluator_prompt

ZmeiGorynych
and
claude
committed
59afd07
annotator: add get_column_sample_values tool and restructure prompts by benchmark flavor

ZmeiGorynych
and
claude
committed
ddca081
docs: document bird-interact-cloud annotate subcommand in README

ZmeiGorynych
and
claude
committed
73435e1
DEV-1525: fix --otf-rebuild no-op + audited gold filename bug found by Codex

ZmeiGorynych
and
claude
committed
988cced
DEV-1525: fix migration collision guard + missing --slayer-setup in scripts

ZmeiGorynych
and
claude
committed
2379fe4
DEV-1525: fix annotation paths + cloud CLI after benchmark rename

ZmeiGorynych
and
claude
committed
7d0d423
DEV-1525: allow oracle mode through _validate_framework_mode

ZmeiGorynych
and
claude
committed
20114ab
DEV-1525: add _validate_framework_mode to cloud CLI; fix old audited gold filename; update README examples

ZmeiGorynych
and
claude
committed
d41702a
DEV-1525: fix _mv dry-run creating parent dirs

ZmeiGorynych
and
claude
committed
058c02c
DEV-1525: remove residual replace('-','_') calls in eval modules

ZmeiGorynych
and
claude
committed
1a3f0b1
DEV-1525: fix gated gold GCS upload + submit validation + annotator autodiscovery

ZmeiGorynych
and
claude
committed
026fcc1

Commits on Jun 4, 2026

annotator: add typed evidence-source prefix enforcement and column-sample-values tool

ZmeiGorynych
and
claude
committed
74a05e8
annotator: fix provenance pre-fill, authoritative masked-term overwrite, n-clamping, dup-variant-id

ZmeiGorynych
and
claude
committed
9f0b6a1
DEV-1525: fix process-reviews round 5 findings

ZmeiGorynych
and
claude
committed
ce6cca4
Merge remote-tracking branch 'origin/egor/dev-1518-annotator-agent' into egor/dev-1525-clean-up-handling-of-multiple-benchmarks
ZmeiGorynych
committed
d33f97e
DEV-1525: fix test_annotator_agent after dev-1518 merge

ZmeiGorynych
and
claude
committed
160bff1
DEV-1525: fix Round-6 review findings

ZmeiGorynych
and
claude
committed
7ccd6e4

Navigation Menu

Uh oh!

DEV-1525: clean up handling of multiple benchmarks - #26

DEV-1525: clean up handling of multiple benchmarks#26
ZmeiGorynych merged 26 commits into
mainfrom
egor/dev-1525-clean-up-handling-of-multiple-benchmarks

Commits on Jun 3, 2026

Merge remote-tracking branch 'origin/egor/dev-1518-annotator-agent' into egor/dev-1525-clean-up-handling-of-multiple-benchmarks

annotator: fix annotation quality gaps found in smoke test

DEV-1525: WIP — clean up handling of multiple benchmarks

Merge remote-tracking branch 'origin/main' into egor/dev-1525-clean-up-handling-of-multiple-benchmarks

Merge remote-tracking branch 'origin/egor/dev-1518-annotator-agent' into egor/dev-1525-clean-up-handling-of-multiple-benchmarks

annotator: fix metadata_evidence string→list and primary boolean validation

annotator: fix masked-term dedup to only block on is_mask=True entries

DEV-1525: clean up handling of multiple benchmarks

DEV-1525: fix migration script self-move bug; update docs/scripts with new benchmark and framework names

annotator: validate solvable tasks require gold variants; insufficient tasks require evaluator_prompt

annotator: add get_column_sample_values tool and restructure prompts by benchmark flavor

docs: document bird-interact-cloud annotate subcommand in README

DEV-1525: fix --otf-rebuild no-op + audited gold filename bug found by Codex

DEV-1525: fix migration collision guard + missing --slayer-setup in scripts

DEV-1525: fix annotation paths + cloud CLI after benchmark rename

DEV-1525: allow oracle mode through _validate_framework_mode

DEV-1525: add _validate_framework_mode to cloud CLI; fix old audited gold filename; update README examples

DEV-1525: fix _mv dry-run creating parent dirs

DEV-1525: remove residual replace('-','_') calls in eval modules

DEV-1525: fix gated gold GCS upload + submit validation + annotator autodiscovery

Commits on Jun 4, 2026

annotator: add typed evidence-source prefix enforcement and column-sample-values tool

annotator: fix provenance pre-fill, authoritative masked-term overwrite, n-clamping, dup-variant-id

DEV-1525: fix process-reviews round 5 findings

Merge remote-tracking branch 'origin/egor/dev-1518-annotator-agent' into egor/dev-1525-clean-up-handling-of-multiple-benchmarks

DEV-1525: fix test_annotator_agent after dev-1518 merge

DEV-1525: fix Round-6 review findings