DEV-1586: pre-encoded mode for the SLayer claude_sdk agents

[ZmeiGorynych]

What

Adds a read-only pre-encoded flavor to the four SLayer claude_sdk OTF agents (claude_sdk_otf, claude_sdk_otf_ainteract, ×v0/v1) so they run against an already-encoded SLayer datasource instead of encoding KB on the fly — mirroring the pydantic_ai committed-reference consumer.

Selected by a new flag --pre-encoded-models {otf,custom}:

• otf (default sense) → the encoding-agent output at slayer_models_otf/<benchmark>/<db>
• custom → the hand-curated slayer_models/<db>
• omitted → encode KB on the fly (unchanged behavior)

In pre-encoded mode the agents have no model-mutation tools (create_model/edit_model/save_memory/validate_models dropped) — introspection only.

Factoring (no copy-pasted "zoo")

Per the brief, the four agents are factored rather than duplicated:

• agents/_pre_encoded.py — source-root selection, a benchmark-aware per-task storage resolver (HARD-8 deleted-KB masking via build_task_variant_storage, threading mini_interact_root/db_root correctly for non-mini benchmarks), fail-clear on a missing reference / embeddings.db, write-tool filtering, slayer_setup derivation.
• agents/_pre_encoded_prompts.py — one introspect-only prompt per eval_mode, reused by v0+v1. The frozen on-the-fly prompts are untouched.
• Each agent gained a pre_encoded_source param that branches storage, tool surface (v0 whitelist + v1 MAIN_TOOLS/subagent), prompt, hooks, and disallowed_tools.

Flag retirement & cloud wiring

• --slayer-setup is retired from both user-facing CLIs; the internal slayer_setup is derived from --pre-encoded-models and kept because cloud artifact routing / fingerprinting / merge consume it. The flag is gated to the claude_sdk SLayer frameworks and to --query-mode slayer.
• Source-aware artifact upload/download/presence routing; job-args + resubmit thread pre_encoded_source (legacy pre-encoded manifests default to custom); provenance recorded in run_metadata / SubmissionConfig / collation; upload-back merge stays gated to otf_encode + on-the-fly so read-only runs skip it.

Tooling

scripts/build_otf_references.py batch-builds the LLM-encoded references for a whole benchmark (the consumer fails clear pointing at it).

Tests

tests/test_dev1586_pre_encoded.py (mechanical contracts only, per project rule) + migration of the slayer-setup CLI tests to the new flag. Also fixes a pre-existing CWD leak of embeddings.db in test_one_shot_run.py via an autouse cwd-isolation fixture. Full non-integration suite: 3341 passed.

Plan and tests were reviewed by Codex (two rounds); all findings folded in.

🤖 Generated with Claude Code

Summary by CodeRabbit

Release Notes

• New Features

  • Added read-only pre-encoded SLayer support for claude_sdk using --pre-encoded-models {otf,custom} (including pre-encoded a-interact flows).
  • Added a batch utility to build encoded SLayer references for benchmark databases.

• Documentation

  • Updated README and example commands to use --pre-encoded-models and reflect that slayer mode now encodes on-the-fly by default.

• Bug Fixes

  • Improved pre-encoded run validation and persisted pre_encoded_source in run metadata.

• Chores

  • Retired the --slayer-setup CLI flag in favor of --pre-encoded-models.

[linear]

DEV-1586 Support pre-encoding with the claude_sdk based agents

Review in Linear

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 6ba3094f-1cb7-406f-881e-c6696a089a8e

📥 Commits

Reviewing files that changed from the base of the PR and between cc34877 and 20f7359.

📒 Files selected for processing (4)

• scripts/run_haiku_slayer.sh
• scripts/run_three_way.sh
• src/bird_interact_agents/cloud/driver.py
• tests/test_dev1586_pre_encoded.py

💤 Files with no reviewable changes (1)

• scripts/run_haiku_slayer.sh

🚧 Files skipped from review as they are similar to previous changes (1)

• tests/test_dev1586_pre_encoded.py

---

📝 Walkthrough

Walkthrough

Introduces a read-only "pre-encoded" SLayer mode (DEV-1586) for four Claude SDK OTF agent variants. Replaces the retired --slayer-setup CLI flag with --pre-encoded-models {otf,custom}, threads pre_encoded_source through the runner, cloud driver, Ray worker, and persistence layers, adds per-agent write-tool stripping and prompt switching, and provides a batch script to build OTF references offline.

Changes

Pre-encoded SLayer mode (DEV-1586)

Layer / File(s)	Summary
Pre-encoded shared module: constants, validation, and storage resolution src/bird_interact_agents/agents/_pre_encoded.py	Defines PRE_ENCODED_SOURCES, WRITE_SLAYER_TOOLS, PreEncodedSetupError, derive_slayer_setup, validate_pre_encoded_source, pre_encoded_source_root, write-tool strip helpers, fail-clear _assert_reference_present, and the async resolve_pre_encoded_storage_dir with HARD-8 deleted-KB masking.
Pre-encoded prompt templates src/bird_interact_agents/agents/_pre_encoded_prompts.py	Assembles read-only tool guidance block, discover-then-query discipline, and _SUBMIT_CONTRACT into two exported constants: SLAYER_PRE_ENCODED_ONE_SHOT and SLAYER_PRE_ENCODED_AINTERACT.
Four Claude SDK agent adapters: pre-encoded mode branching src/bird_interact_agents/agents/claude_sdk_otf/agent.py, src/bird_interact_agents/agents/claude_sdk_otf_v1/agent.py, src/bird_interact_agents/agents/claude_sdk_otf_ainteract/agent.py, src/bird_interact_agents/agents/claude_sdk_otf_ainteract_v1/agent.py	Each adapter gains pre_encoded_source constructor parameter, slayer_setup consistency enforcement, per-task storage branching to resolve_pre_encoded_storage_dir, prompt-template switching, write-tool stripping from MCP tool surface, conditional write-filter hook exclusion, and updated disallowed_tools/PreToolUse hook wiring.
Runner: pre_encoded_source derivation, validation, and CLI flag replacement src/bird_interact_agents/run.py	Adds _PRE_ENCODED_FRAMEWORKS, rewrites _validate_slayer_setup to enforce source/framework/mode consistency, updates make_runner/run_one_task/run_evaluation to derive slayer_setup from pre_encoded_source, wires pre_encoded_source into all agent constructors, replaces --slayer-setup with --pre-encoded-models.
Cloud pipeline and persistence: pre_encoded_source plumbing src/bird_interact_agents/cloud/cli.py, src/bird_interact_agents/cloud/driver.py, src/bird_interact_agents/cloud/ray_app.py, src/bird_interact_agents/cloud/collation.py, src/bird_interact_agents/eval/annotation_schema.py, src/bird_interact_agents/results_db.py	Cloud CLI derives slayer_setup from the new flag; driver manifests emit pre_encoded_source, branch slayer_models_otf vs slayer_models uploads, and validate per-DB embeddings.db readiness; Ray app threads pre_encoded_source through run_pool to workers; results_db adds a nullable pre_encoded_source column with backward-compatible migration; SubmissionConfig gains the optional field.
Batch OTF reference builder script scripts/build_otf_references.py	New script enumerates unique DB names from a benchmark data file, builds the Pydantic-AI model with optional Anthropic cache settings, and calls ensure_db_reference idempotently for each DB (or a --only subset).
Test suite: pre-encoded mode contract, CLI flag, and agent wiring tests/test_dev1586_pre_encoded.py, tests/test_slayer_setup_flag.py, tests/test_claude_sdk_otf_v1_run_wiring.py, tests/test_claude_sdk_otf_ainteract_v1_run_wiring.py, tests/test_one_shot_mode.py, tests/test_one_shot_run.py, tests/test_pydantic_ai_otf_encode_run_wiring.py, tests/cloud/test_cli.py, tests/cloud/test_dev1555_stage2_cli.py	New test_dev1586_pre_encoded.py covers the full pre-encoded contract; existing test helpers drop --slayer-setup from argv and replace assertions with --pre-encoded-models/pre_encoded_source equivalents; test_one_shot_run.py gains a CWD isolation autouse fixture.
README and scripts: default OTF encoding and pre-encoded mode documentation README.md, scripts/run_haiku_slayer.sh, scripts/run_three_way.sh	Removes --slayer-setup on-the-fly from all example commands, adds "Pre-encoded mode (read-only, DEV-1586)" section documenting --pre-encoded-models {otf,custom}, and clarifies on-the-fly as the new default.

Sequence Diagram(s)

sequenceDiagram
  participant CLI as run.py CLI
  participant Validator as _validate_slayer_setup
  participant Runner as make_runner
  participant Agent as ClaudeSDKOtfAgent
  participant PreEncoded as _pre_encoded
  participant SLayerMCP as SLayer MCP Server

  CLI->>CLI: parse --pre-encoded-models otf → pre_encoded_source="otf"
  CLI->>CLI: derive_slayer_setup("otf") → slayer_setup="pre-encoded"
  CLI->>Validator: _validate_slayer_setup(framework, query_mode, slayer_setup, pre_encoded_source)
  Validator-->>CLI: ok
  CLI->>Runner: run_evaluation(..., pre_encoded_source="otf", slayer_setup="pre-encoded")
  Runner->>Agent: __init__(..., pre_encoded_source="otf", slayer_setup="pre-encoded")
  Agent->>Agent: validate_pre_encoded_source("otf")
  Note over Agent: run_task per DB
  Agent->>PreEncoded: resolve_pre_encoded_storage_dir(db, task_data, benchmark, "otf")
  PreEncoded->>PreEncoded: _assert_reference_present("otf", ref_dir)
  PreEncoded-->>Agent: (slayer_storage_dir, deleted_kb_ids)
  Agent->>Agent: strip_write_slayer_tools(SLAYER_MCP_TOOLS)
  Agent->>Agent: use SLAYER_PRE_ENCODED_ONE_SHOT prompt
  Agent->>SLayerMCP: list_datasources, search, inspect_model, query (read-only)

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• MotleyAI/bird-interact-agents#23: The batch scripts/build_otf_references.py script directly reuses the DEV-1454 on-the-fly reference-build primitives (ensure_db_reference, setup-encoder wiring).
• MotleyAI/bird-agents#2: Modifies the same cloud slayer artifact plumbing (cloud/cli.py, cloud/driver.py, cloud/ray_app.py) that this PR extends with pre_encoded_source selection between slayer_models_otf and slayer_models.
• MotleyAI/bird-agents#40: Both PRs extend results_db.insert_run_metadata and open_db with additional nullable diagnostic columns and backward-compatible ALTER TABLE migration.

Poem

🐇 Hop hop, the encoder's done its work at last,
No need to re-carve each KB as you pass!
--pre-encoded-models otf is the key,
Read-only SLayer tools, as quiet as can be.
The write tools are stripped, the references sealed tight,
A pre-built rabbit burrow — pure delight! 🌿

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 44.85% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The PR title clearly identifies the primary change: adding pre-encoded mode support to SLayer claude_sdk agents, matching the core feature described in pr_objectives. The title is specific, concise, and accurately reflects the main objective.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}