Skip to content

fix(ci): re-baseline storyboard floors to measured main #6885

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
bokelley wants to merge 2 commits into from
+21 −13
Closed

fix(ci): re-baseline storyboard floors to measured main #6885

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
78b208e
fix(ci): re-baseline storyboard floors to measured main
bokelley Aug 25, 2026
d1af199
test: align floor mirror guard with re-baselined values
bokelley Aug 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/recover-storyboard-floors.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"adcontextprotocol": patch
---

Re-baseline the storyboard conformance floors to the values measured on main's own CI run after the #6815 coverage rebaseline set them above reality (creative 48 clean/200 steps, creative-builder 48/172, sales 126/553). Every gap below the prior floors is authored known-failing skips with zero step failures; the workflow had been red on main and the mirrored local pre-push gate blocked all compliance-source pushes.
10 changes: 5 additions & 5 deletions .github/workflows/training-agent-storyboards.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,12 +85,12 @@ jobs:
min_passing_steps: 157
- surface: current
tenant: creative
min_clean_storyboards: 49
min_passing_steps: 209
min_clean_storyboards: 48
min_passing_steps: 200
- surface: current
tenant: creative-builder
min_clean_storyboards: 50
min_passing_steps: 184
min_clean_storyboards: 48
min_passing_steps: 172
- surface: current
tenant: brand
min_clean_storyboards: 45
Expand Down Expand Up @@ -625,7 +625,7 @@ jobs:
# three signed-request route profiles; storyboard applicability is
# reported separately in the aggregate summary.
MIN_CLEAN: 126
MIN_PASSED: 556
MIN_PASSED: 553

@aao-secretariat aao-secretariat Bot Aug 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MUST FIX: Third floor mirror not updated. tests/run-storyboards-sharding.test.cjs asserts the old literals against both files this PR edits and will go red on merge: MIN_PASSED: 556 (L259), "sales:126:556" (L260), and the baselines table ['sales', 126, 556], ['creative', 49, 209], ['creative-builder', 50, 184] (L293/295/296). The PR body says it sets "both floor mirrors" — there are three. Update the test to 553 / 200 / 172 (creative 48, creative-builder 48) or the suite fails. The end-to-end matrix run validated the storyboard run, not these literal assertions.

run: |
set -euo pipefail
echo "Clean storyboards: ${CLEAN} (floor: ${MIN_CLEAN})"
Expand Down
9 changes: 6 additions & 3 deletions scripts/run-storyboards-matrix.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -253,14 +253,17 @@ if [ "${FLOOR_SET}" = "3.0-compat" ]; then
)
else
TENANTS=(
# Re-measured on main@32842430342 (post-#6815 coverage rebaseline): every
# gap below the prior floors was authored known-failing skips, zero step
# failures — see #6876.
# Ratcheted from the first capability-resolved replay. The runner reports
# declared-scope applicability and quarantines separately from these
# clean-result-row and passing-step regression floors.
"signals:45:80"
"sales:126:556"
"sales:126:553"
"governance:47:157"
"creative:49:209"
"creative-builder:50:184"
"creative:48:200"
"creative-builder:48:172"
"brand:45:116"
"si:42:50"
)
Expand Down
10 changes: 5 additions & 5 deletions tests/run-storyboards-sharding.test.cjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -256,8 +256,8 @@ test('current /sales runs fixed orchestrators with isolated children behind one
assert.match(workflow, /needs: sales_storyboard_orchestrators/);
assert.match(workflow, /ORCHESTRATOR_RESULT: \$\{\{ needs\.sales_storyboard_orchestrators\.result \}\}/);
assert.match(workflow, /MIN_CLEAN: 126/);
assert.match(workflow, /MIN_PASSED: 556/);
assert.match(matrixRunner, /"sales:126:556"/);
assert.match(workflow, /MIN_PASSED: 553/);
assert.match(matrixRunner, /"sales:126:553"/);
assert.match(workflow, /Training agent · current \/sales/);
assert.match(workflow, /echo "failed=\$\{failed_sum\}"/);
assert.match(workflow, /echo "not_applicable=\$\{not_applicable_sum\}"/);
Expand Down Expand Up @@ -290,10 +290,10 @@ test('current training-agent floors are ratcheted and mirrored by local and CI r
const matrixRunner = fs.readFileSync(MATRIX_RUNNER, 'utf8');
const baselines = [
['signals', 45, 80],
['sales', 126, 556],
['sales', 126, 553],
['governance', 47, 157],
['creative', 49, 209],
['creative-builder', 50, 184],
['creative', 48, 200],
['creative-builder', 48, 172],
['brand', 45, 116],
['si', 42, 50],
];
Expand Down
Loading