Article data-collection: vectors x models x switches matrix runner emitting markdown table

[nabinpkl]

Status: open, deferred until #B lands. Article-Phase-3. Produces the data table for the engineering post on prompt-injection robustness across models.

Why

The bug fix in #33 showed that today's free-tier primary model (nvidia/nemotron-3-super-120b-a12b:free) failed text-vs-tool divergence on roleplay and system-tag-spoofing vectors WHEN the boundary rail and constitution gate were off. The system prompt told the model to refuse these, but the model didn't. The article's thesis: "writing rules in the system prompt is necessary but not sufficient; here's how often each model actually follows its own rules without enforcement, vector by vector."

To produce the data, we need to run the eval matrix:

• N vectors (today: base64, roleplay, system-tag-spoofing, off-domain, plain prompt-injection; ~5)
• M models (sample free-tier OpenRouter models that vary in size, family, instruction tuning; ~5-8)
• K switch combinations (full defenses, defenses off, per-defense ablations; ~8)

= 5 x 6 x 8 = 240 cells. Free-tier OpenRouter at 20 RPM means ~12 minutes per pass through the matrix; sequential execution is fine.

Scope

New module agent-service/src/agent_service/scripts/injection_ablation.py:

• Reads a YAML config naming the model list, the suite paths, and the switch combinations to test.
• For each (model x suite x switches) cell:
  1. Sets AGENT_PRIMARY_MODEL env var to the cell's model.
  2. Restarts the agent service container (or hot-reloads the model client; whichever is faster).
  3. Runs the suite via just eval with overrides for the switch state.
  4. Captures pass/fail per probe per case.
• Aggregates into a markdown table:

| Vector | Defenses-on baseline | Model A defenses-off | Model B defenses-off | ... |
|---|---|---|---|---|
| base64 | 12/12 PASS | 8/12 (67%) | 11/12 (92%) | ... |
| roleplay | 12/12 PASS | 4/12 (33%) | 12/12 (100%) | ... |
| ... |

Plus a per-cell trace-id so the article can deep-link into Langfuse for the qualitative narrative example.

• Output: evals/articles/injection_ablation_2026-XX-XX.md (date-stamped) + a JSON sidecar with raw probe results per cell.

Out of scope

• Statistical significance / multi-run averaging. Single run per cell is fine for an exploratory article. If a cell flaps, the article's qualitative analysis (Langfuse trace deep-links) covers it.
• A web UI for the table. The markdown is the deliverable.
• Cost tracking per cell. Free tier; not a concern.

Done when

• python -m agent_service.scripts.injection_ablation --config evals/articles/injection_ablation_config.yaml --out evals/articles/injection_ablation_<date>.md runs the full matrix and emits the table + JSON sidecar.
• The article's data table is reproducible from the script + a fixed config + a fixed primitive snapshot.
• README addition documents the script alongside just eval.

Refs

• #33 (the bug that motivated the article).
• #A (StayInRoleSwitches scaffolding; required to toggle per-defense switches per cell).
• #B (prompt modularization; required for per-rule ablation cells).
• Agent-SafetyBench (arxiv 2412.14470, Dec 2025) for the text-vs-tool divergence framing.
• Confident AI: Multi-Turn LLM Evaluation in 2026 (2026-03-22) for related multi-turn extension (Multi-turn eval coverage: knowledge retention, context drift, goal hijacking #31).

[nabinpkl]

Cross-ref note: this ticket is #C in the previous discussion; #A is #35, #B is #36.