Analyze SLayer query-syntax rejections in mini-interact slayer runs

[ZmeiGorynych]

What

Quantifies how often the agent's SlayerQuery DSL payloads get rejected in slayer mode, what it gets wrong, and whether that friction ever sinks a task.

• scripts/analyze_query_syntax_rejections.py — importable library + CLI. Selects the latest slayer *.trajectory.json per (db, iid), pairs each query-tool call with its result by tool_use_id, and classifies every result via one ordered regex taxonomy into dsl (headline) / semantic / gate / infra / other / ok. Computes the DSL failure-mode taxonomy, retry-to-recovery chains, and major-contributor-to-failure flags (direct / never_clean / budget_burn). --json for the payload, plain text for a terminal summary.
• notebooks/query_syntax_rejections.ipynb — the reproducible human-readable view (matplotlib plots + pandas tables), including a detailed per-mode breakdown of every DSL failure mode with real example error text + plain-English descriptions.
• analysis pyproject extra (matplotlib/pandas/jupyter/nbformat/nbconvert) for the notebook tooling, kept out of all so runtime installs stay lean.
• Tests pinning the taxonomy buckets, retry-chain logic, the three contributor flags, latest-trajectory selection, per-sub_tag example capture/dedup, description coverage, and a drift guard tying the budget literals to harness.ACTION_COSTS.

Scope

Headline = DSL-validation rejections only (SLayer's query compiler refused the spec). Excluded but tallied for context: semantic (DB-execution failures — no such column/table, generated-SQL syntax error), gate (submit_query without previewing via query first), and infra (permission-mode denials, schema-drift — harness artefacts).

Findings (298 latest slayer tasks, 4373 query-tool calls)

1. Failure modes (399 DSL rejections): order_shape dominates at ~46% (writing order as {"col":"desc"} instead of {"column":"col","direction":"desc"}). The non-order_shape tail (~54%, 15 modes) clusters into filter composition, aggregation/measure idioms, and name/shape resolution.
2. Retries to recovery: ~85% of chains fixed on the very next attempt; 1 unrecovered chain in the whole dataset.
3. Major contributor to failure? No — all flags zero. Every one of the 44 failures is a wrong_result (semantic miss), the agent always recovered syntactically, and the high patience budget (~1010 coins) meant ≤1-coin retries never came close to exhaustion.

Takeaway: DSL friction is a prompt/ergonomics opportunity (fixing the order shape alone removes ~46% of rejections), not a driver of failures in these runs.

Test plan

• env -u SSH_AUTH_SOCK uv run --extra all --extra dev --extra pydantic-ai pytest → 2795 passed, 94 skipped.
• Notebook re-executes cleanly end-to-end (0 errors, 5 embedded plots + per-mode markdown breakdown).
• uv run python scripts/analyze_query_syntax_rejections.py --mode slayer reproduces the headline numbers above.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added optional analysis dependency extra with notebook tools for analysis workflows.
  • Added query syntax rejection analysis script for benchmarking workflows.

• Tests

  • Added test suite validating analysis functionality, error classification, and retry patterns.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds scripts/analyze_query_syntax_rejections.py, a new dual-mode module that reads benchmark trajectory JSON files, classifies query-tool results via a regex taxonomy into DSL/semantic/gate/infra/ok buckets, chains DSL retries, reconstructs budget signals, flags major-contributor tasks, and outputs either a JSON payload or terminal summary. A pytest suite and a new analysis optional dependency extra are also added.

Changes

Query Syntax Rejection Analyzer

Layer / File(s)	Summary
Module constants and rejection taxonomy scripts/analyze_query_syntax_rejections.py	Defines QUERY_TOOLS, ERROR_TAXONOMY, DSL_SUBTAGS, SUBTAG_DESCRIPTIONS, and regex constants (_FILE_MODE_RE, _TS_RE, _BUDGET_NOTE_RE) for trajectory selection, budget parsing, and ordered regex-based bucket/subtag classification.
Trajectory selection and tool-result extraction scripts/analyze_query_syntax_rejections.py	Implements mode_from_filename, latest_trajectory_per_iid (selects newest file per (db, iid)), and extraction/classification helpers that walk assistant/user message pairs to produce a typed (bucket, subtag, text, tool_name) sequence.
Per-task DSL chains, budget signal, and analyze_trajectory scripts/analyze_query_syntax_rejections.py	_dsl_chains groups consecutive DSL results into recovered/unrecovered chains; _budget_signal reconstructs remaining/total budget from tool-result text; analyze_trajectory combines these into per-task bucket counts, subtag counts, example texts, chain stats, and major_contributor flags (never_clean, direct, budget_burn).
Aggregation, build_payload, render_text, and CLI scripts/analyze_query_syntax_rejections.py	aggregate rolls up per-task records into headline counts, DSL breakdowns, retry histogram, and sorted major-contributor list. build_payload orchestrates end-to-end JSON output. render_text formats a terminal summary. main exposes --benchmark, --mode, and --json flags.
Test suite and analysis extra tests/scripts/test_analyze_query_syntax_rejections.py, pyproject.toml	Pytest suite pins classify_result precedence, _dsl_chains recovery logic, analyze_trajectory flag conditions, latest_trajectory_per_iid file selection, build_payload end-to-end aggregates, DSL example deduplication/capping, SUBTAG_DESCRIPTIONS completeness, and ACTION_COSTS drift guards. pyproject.toml adds the analysis optional extra with matplotlib, pandas, jupyter, nbformat, nbconvert.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Poem

🐇 Hoppity-hop through trajectory trails,
Regex taxonomies sorting details,
DSL chains counted, budgets surveyed,
"major_contributor" flags proudly displayed!
The bunny now knows which queries failed first —
and renders the stats to quench your thirst. 🥕

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 34.29% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'Analyze SLayer query-syntax rejections in mini-interact slayer runs' directly and clearly summarizes the main contribution: introducing analysis tooling to quantify DSL payload rejections during slayer task execution.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (1)

scripts/analyze_query_syntax_rejections.py (1)

781-784: 💤 Low value

Consider tracking skipped trajectories for observability.

Silently continuing on parse errors could mask data issues (e.g., truncated files, encoding problems). Adding a counter or debug log would help diagnose unexpected results without breaking the best-effort analysis.

+    skipped = 0
     for (db, iid), path in sorted(chosen.items()):
         try:
             data = json.loads(path.read_text())
         except (json.JSONDecodeError, OSError):
+            skipped += 1
             continue
         per_task.append(analyze_trajectory(data, db=db, iid=iid))
     return {
         "benchmark": benchmark,
         "mode": mode,
         "per_task": per_task,
         "aggregate": aggregate(per_task),
+        "skipped_files": skipped,
     }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/analyze_query_syntax_rejections.py` around lines 781 - 784, The
try-except block that catches json.JSONDecodeError and OSError exceptions when
reading and parsing JSON files is silently continuing without any logging or
tracking. Add instrumentation to log or count skipped trajectories when these
exceptions occur in the exception handler. This could be implemented by adding a
debug log statement that includes the file path and exception details before the
continue statement, or by maintaining a counter of skipped files. This will help
diagnose data issues like truncated or malformed files without breaking the
best-effort analysis flow.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@scripts/analyze_query_syntax_rejections.py`:
- Around line 781-784: The try-except block that catches json.JSONDecodeError
and OSError exceptions when reading and parsing JSON files is silently
continuing without any logging or tracking. Add instrumentation to log or count
skipped trajectories when these exceptions occur in the exception handler. This
could be implemented by adding a debug log statement that includes the file path
and exception details before the continue statement, or by maintaining a counter
of skipped files. This will help diagnose data issues like truncated or
malformed files without breaking the best-effort analysis flow.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 83c1fcf8-251b-4758-9fd6-f9662301e36f

📥 Commits

Reviewing files that changed from the base of the PR and between ecd2b29 and 5711049.

⛔ Files ignored due to path filters (1)

• uv.lock is excluded by !**/*.lock

📒 Files selected for processing (4)

• notebooks/query_syntax_rejections.ipynb
• pyproject.toml
• scripts/analyze_query_syntax_rejections.py
• tests/scripts/test_analyze_query_syntax_rejections.py