Storyboards: shallow validation gap on read tasks + missing storyboards for 7 tasks

[bokelley]

Problem

Current storyboard validations on read/list tasks are essentially schema + correlation_id echo. Across all field_value checks in static/compliance/source/specialisms/:

• ~119/125 are path: \"context.correlation_id\" echoes
• Only ~6 inspect actual response content (compliant, verdict, deployments[0].type, signal_id.source, success, deleted, status, brand_id)

A seller can return wrong, empty, or unrelated data and pass every read storyboard.

Examples of the gap

• get_media_buys is exercised in 6 storyboards (sales-broadcast-tv, sales-guaranteed, sales-non-guaranteed, protocols/media-buy, get-media-buys-pagination-integrity, idempotency). None assert that media_buys[0].media_buy_id == \$context.media_buy_id from the prior create_media_buy. The seller can return any media buy.
• get_creative_delivery (creative-ad-server) — schema + correlation_id only. No assertions on impressions, spend, variants, or media-buy linkage.
• get_media_buy_delivery — same shallow pattern across 5 specialisms.
• Lists (list_creatives, list_property_lists, list_collection_lists, list_content_standards, list_accounts, list_creative_formats) — schema + correlation_id; never asserts the just-created entity appears.

Two tracks

Track A — Data-grounded validations on existing read steps

Add cross-step assertions where state is already threaded through \$context.*:

• get_media_buys / get_media_buy_delivery — assert returned media_buy_id matches the captured one
• get_creative_delivery — assert returned creative_ids include the synced ones; impressions ≥ 0; spend ≤ buy budget
• list_* after a create_* — assert the created id appears in the list response
• validate_property_delivery is the existing reference pattern (asserts compliant: false based on input shape) — generalize that approach

Most of these need the cross-step comparison primitive in #2642 (field_equals_context, field_subset_of_context, etc.). Track A is partially blocked on that schema work.

In-step assertions that don't need new primitives (and can land now):

• field_present on substantive fields, not just top-level keys (e.g., media_buys[0].packages[0].status, delivery.totals.impressions)
• field_value for known-deterministic fields (e.g., when the prior step seeded a fixture, the read should return the seeded brand domain)

Track B — Missing storyboards for 7 tasks

These tasks exist in static/schemas/source/**/*-request.json but appear in zero storyboard steps:

Task	Domain	Notes
search_brands	brand	Brand discovery via natural language; no coverage
get_creative_features	creative	Semantic feature extraction; no coverage
get_media_buy_artifacts	media-buy	IO/PDF artifact retrieval; no coverage
update_rights	brand	Modify acquired rights; no coverage
creative_approval	creative	Review callback; no coverage
tasks_get	core	Async polling — no end-to-end working→submitted→completed storyboard
tasks_list	core	Same

identity_match / context_match (TMP) are intentionally out of scope here — separate track.

Cross-cutting gaps surfaced during the skim

1. No state-roundtrip assertions — IDs threaded into \$context are never validated as the response payload (depends on Storyboard schema: cross-step comparison validation primitive #2642).
2. No quantitative invariants — "spend ≤ budget", "impressions ≥ 0", "totals == sum of packages", "flight_dates within campaign window" are nowhere.
3. No async polling storyboard — tasks/get is only meaningful after a working/submitted response and there's no storyboard exercising it.
4. Read-task error paths thin — governance-denied is well covered, but auth/not_found/validation errors on read tasks are largely absent.

Suggested sequencing

1. Land Storyboard schema: cross-step comparison validation primitive #2642 (cross-step comparison primitive).
2. Track A pass: add data-grounded validations to existing read steps using the new primitives.
3. Track B: add storyboards for the 7 missing tasks (each as its own PR; tasks_get/tasks_list likely as one async-polling storyboard under protocols/).
4. Quantitative invariants pass (separate, may need new check kinds for arithmetic).

Related

• Storyboard schema: cross-step comparison validation primitive #2642 — cross-step comparison primitive (prerequisite for most of Track A)
• storyboard: dependency-impact end-to-end scenario #2860 — dependency-impact end-to-end scenario
• Docs gap: storyboards vs. scenarios test surfaces are not disambiguated #4035 — storyboards vs. scenarios surface disambiguation

[bokelley]

Triage

Classification: Compliance storyboard gap — additive harness work, no schema or protocol changes required for the unblocked subset.

Bucket: compliance-suite

Expert panel: ad-tech-protocol-expert + code-reviewer

Status: drafting-pr → PR #4294 opened (draft, targeting 3.0.x)

---

What shipped in PR #4294

Three Track B tasks now have storyboard coverage, added as new phases on existing storyboards (no existing steps or validations touched):

Task	Storyboard	Notes
search_brands	protocols/brand/index.yaml — new brand_discovery phase	requires_tool: search_brands at step level → non-adopting agents get missing_tool, not failure. Not added to storyboard-level required_tools (would gate GA compliance for an experimental task).
get_creative_features	specialisms/content-standards/index.yaml — new creative_feature_extraction phase	Sample request uses correct assets shape: named-key object with asset_type discriminator per creative-manifest.json.
get_media_buy_artifacts	specialisms/content-standards/index.yaml — new media_buy_artifacts_retrieval phase	controller_seeding: true added to prerequisites; fixtures.media_buys block seeds content_standards_artifacts_mb_1 before the step runs.

---

What remains and why

tasks_get / tasks_list — Needs a placement decision:

• Placing under protocols/core/ requires adding "core" to adcp-protocol.json enum. That's a minor semver bump (new enum value) → incompatible with 3.0.x patch line.
• Alternative: place under an existing protocol storyboard (e.g. protocols/buyer/) and use comply_scenario: async_task_polling. This avoids the enum change but is semantically awkward.
• Also needs a force_task_completion controller scenario — not currently defined.
• Decision needed: accept minor bump path, or find a patch-compatible placement.

creative_approval — Architectural gap in the harness:

• This is a buyer-initiated HTTP POST to the brand agent's approval_webhook URL. The buyer is the HTTP server receiving the callback — not the MCP client making tool calls.
• No harness primitive exists for "buyer-as-HTTP-server receives inbound POST." Needs a new primitive (e.g. webhook_receive step kind) before this can be storyboarded.

update_rights — Two open questions:

• x-status: experimental — confirm this belongs on 3.0.x or needs a feature branch.
• The task modifies existing rights state; needs a fixture/seeding pattern for the rights record to update.

Track A (cross-step assertions) — All items blocked on issue #2642. The field_equals_context / field_subset_of_context comparison primitives are required to assert that a brand_id returned in one step matches the value used in a subsequent step. No PR exists for #2642.

---

Blocked-on: #2642 (Track A cross-step assertions)

---

Generated by Claude Code

[bokelley]

/triage execute

PR #4294 closed due to bad rebase against 3.0.x (2.2M lines of merge fallout). Redo as a fresh PR against main (not 3.0.x). Original scope intact:

• Add storyboard coverage for search_brands, get_creative_features, get_media_buy_artifacts as new phases on existing storyboards.
• Use requires_tool: at step level so non-adopting agents get missing_tool, not failure.
• Do NOT add to storyboard-level required_tools (would gate GA compliance for experimental tasks).