feat: scripts/cascade_for_combo.py — per-combo cascade stats - #48
Merged
Conversation
Adds a script that, given (benchmark, agent_model, mode), walks
runs/<benchmark>/, joins each annotation against its cloud manifest
(results/<benchmark>/cloud/<id>/manifest.json, with a GCS fallback that
caches locally), and prints the full cascade in two views:
* cumulative N1..N9 — monotone, with Δ vs prev (N1 is the headline
original-gold pass rate the user keeps asking for);
* mutually-exclusive partition L1..L11.
Two non-obvious rules baked in:
* latest-per-task uses annotated_at but SKIPS verdict=="eval_failed"
so a stale regrade whose grader infrastructure choked doesn't
override the genuine earlier verdict;
* model match is a case-insensitive substring (`opus` matches
`anthropic/claude-opus-4-7`) so the operator doesn't have to type
the full registry id.
The existing aggregate_cascading_latest in eval/cascading_report.py
applies neither filter, hence the new script rather than a flag.
Drive-by: fixes test_merge_no_overwrite_if_present, which was passing
only because a leftover runs/mini-interact/alien/alien_1/r1.json (from
another test's pollution) happened to exist. Now sandboxes BIRD_RUNS_ROOT
and pre-creates at the actual DEV-1533 dest path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
|
Warning Review limit reached
More reviews will be available in 2 hours, 27 minutes, and 33 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughAdds Changescascade_for_combo CLI and tests
Annotation destination path layout fix
Sequence Diagram(s)sequenceDiagram
participant CLI as main()
participant Collector as collect_latest_per_task
participant LoadManifest as load_manifest
participant LocalFS as Local FS (runs/)
participant GCS as GCS bucket
participant Aggregator as aggregate
participant Renderer as render
CLI->>Collector: benchmark, mode, agent_model
loop each annotation JSON
Collector->>LoadManifest: benchmark, run_id
LoadManifest->>LocalFS: read manifest.json
alt present locally
LocalFS-->>LoadManifest: dict
else allow_gcs=True
LoadManifest->>GCS: download blob
GCS-->>LoadManifest: bytes
else allow_gcs=False
LoadManifest-->>Collector: None (skip run)
end
LoadManifest-->>Collector: manifest dict
Collector->>Collector: filter by mode + model_matches
Collector->>Collector: group by (db, instance_id), pick latest non-eval_failed
end
Collector-->>CLI: chosen paths + counters
CLI->>Aggregator: chosen paths
Aggregator-->>CLI: n_counts + p_counts
CLI->>Renderer: benchmark, mode, agent_model, agg, counters
Renderer-->>CLI: formatted table string
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Comment |
…ed ADC main() eagerly built a google.cloud.storage.Client at startup, which raises DefaultCredentialsError on machines without ADC even when every needed manifest is already on disk under results/<benchmark>/cloud/<run_id>/. Drop the eager construction — gcs.read_manifest already lazily builds a client on first fallback, and load_manifest's except-Exception clause already swallows credential errors as "skip this run". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…invalid→eval_failed migration
Two adjacent correctness bugs in collect_latest_per_task:
(1) When every run for a task had verdict=eval_failed, the
`(real or recs)[-1]` fallback added the latest eval_failed row to
the chosen list, contradicting the docstring's "SKIPPING runs whose
evaluation.verdict == 'eval_failed'" contract and silently miscounting
grader-infrastructure failures as L11 hard fails. Now: omit
eval_failed-only tasks and record them under a new
`skipped_eval_failed_only` counter (also surfaced in the human render).
(2) SubmissionAnnotation._migrate_invalid_verdict upgrades legacy
`verdict="invalid"` + `failure_classification.primary="other"` to
`"eval_failed"` on read, but the selection logic read raw JSON and
only compared against the unmigrated string — so legacy infra
failures could be picked as the latest "real" verdict, defeating
the central skip rule. Mirror the migration in a small
`_effective_verdict` helper at the JSON-read boundary.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ZmeiGorynych
added a commit
that referenced
this pull request
Jun 19, 2026
…ark-runs Reconciles this branch's DEV-1555 / DEV-1561 + CR r1 unification work with PR #46 (DEV-1550 SLayer compact-by-default), PR #48 (cascade_for_combo), PR #50 (query-syntax-rejection analysis), and DEV-1545 + DEV-1546 prompt additions on origin/main. Conflict resolutions: * ``_shared_otf_prompts.py`` — replaced HEAD's V1 helper section with origin/main's (it carries the new ``_DEDUP_VS_RAW_ROWS``, ``_TABLE_SET_PROBE``, ``_GRADER_ZERO_VS_ONE_DIAGNOSTIC``, ``_SLAYER_TOOLS_BLOCK`` etc. plus DEV-1550 ModelColumn / memory drill- in paragraphs), restored our ``_AFTER_REJECTED_DISCIPLINE`` (DEV-1555 stage-2), and kept our four ``*_V0`` snapshots appended at the bottom. Stripped lingering ``query_nested`` mentions from the V1 helpers so the unified-tool contract holds on both sides. * ``claude_sdk/agent.py`` — kept the unified ``query`` schema (``source_model`` + projection fields OR ``queries`` array; ``required: []``) and the runtime ``source_model XOR queries`` gate. The handler now builds the SlayerQuery JSON internally and forwards it to origin/main's DEV-1546 ``query_impl(query_json: str, …)`` for single-stage, or to ``query_nested_impl(queries=…)`` for the nested-DAG form — matching ``submit_query``'s pattern. Picked up origin/main's ``distinct_dimension_values`` field as an additional schema property so the DEV-1546 dim-only auto-dedup opt-out stays reachable through the unified shape. * ``eval/autopsy.py`` — combined origin/main's 2-attempt corrective- retry loop (sends Pydantic validation errors back as a ``tool_result`` so the model self-fixes the archeology_10 regression) with our DEV-1555 model-aware client (``_build_anthropic_client(model)`` + ``requires_thinking(model)`` thinking mode + auto tool_choice for Moonshot/Kimi) and the JSON-text fallback for third-party endpoints that don't honor forced tool_choice. The retry only fires when the model used the tool (the JSON-text-fallback path has no tool_use_id to bind the corrective ``tool_result`` to). * ``run.py._per_task_timeout_s`` — origin/main flipped the default to ``_DEFAULT_PER_TASK_TIMEOUT_S = 0.0`` (no cap). Adjusted our DEV-1555 grace logic so the runaway grace is only added when the operator explicitly opted in to a positive cap — the default-uncapped contract now holds. * V0 prompts kept as thin re-exports from ``_shared_otf_prompts.V0`` snapshots (origin/main had touched ``claude_sdk_otf*/prompts.py`` with DEV-1545/1550 helper composition, but our side reduced those files to one-line re-exports so the V0 snapshot is the single source of truth for the v0 surface). * ``test_shared_otf_prompts.py`` — re-baselined the V1 SHA pins to ``3fa05ac2…``/``65b8eb05…`` → final post-cleanse ``e671aea3…``/``a3fd695c…`` after stripping lingering ``query_nested`` mentions from the merged V1 helpers. * ``test_dev1534_query_wrapper.py`` — rewrote the schema pin to match the unified shape (``source_model``, ``queries``, ``required: []``); the prior DEV-1546 ``query_json``-only pin is superseded. * ``test_dev1546_distinct_dim_values.py`` — repointed the dedup composition tests at the ``SLAYER_OTF_*_V0`` snapshots (where the full ``_DEDUP_VS_RAW_ROWS`` body is inlined byte-for-byte); the v1 prompts teach the same guidance in a shorter form not assembled via the live constant. Rewrote the query-tool-schema test to assert the unified-shape surface. * ``test_dev1555_query_unified_schema.py`` — updated the mock ``query_impl`` to take ``query_json: str`` positional + kwargs (matches DEV-1546's signature) and parse it to verify the wrapper built the right SlayerQuery dict. Full non-integration suite: 3212 passed, 94 skipped, 50 deselected, 0 failed.
This was referenced Jun 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
scripts/cascade_for_combo.py: given(--benchmark, --mode, --agent-model), walksruns/<benchmark>/<db>/<iid>/, joins each annotation against the cloud manifest (local cache → GCS fallback), picks the latest non-eval_failedrun per task, and prints BOTH the cumulative N1..N9 table (with Δ vs prev) and the mutually-exclusive L1..L11 partition.--jsonfor machine output;--no-gcsto disable the fallback.tests/cloud/test_fetch_annotation_merge.py::test_merge_no_overwrite_if_present, which was passing only because a leftoverruns/mini-interact/alien/alien_1/r1.json(from another test's pollution) happened to exist. Now sandboxesBIRD_RUNS_ROOTand pre-creates at the actual DEV-1533 dest path.CLAUDE.mdsection pointing at the script and naming its two non-obvious rules.Why a new script, not a flag on
aggregate_cascading_latestThe existing
aggregate_cascading_latestineval/cascading_report.pyfilters by neither(mode, agent_model)noreval_failed. Both filters matter:verdict=="eval_failed",primary=="other") and shouldn't override the genuine earlier verdict. Hits 36 raw + 11 slayer mini-interact tasks today.rglob+ manifest reads every time.Live numbers (post-fix)
mini-interact / opus / slayermini-interact / opus / rawTest plan
tests/scripts/test_cascade_for_combo.pypin: mode+model filter, eval_failed-override rule, missing-manifest-skip, partition aggregation.uv run python scripts/cascade_for_combo.py --benchmark mini-interact --mode slayer --agent-model opusreturns the table above.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation
Tests