ci: a group that never cancels must never queue, or GitHub discards the baseline (#274) - #1323
Merged
Conversation
…ict (#274) The scheduled lane publishes a verdict for 37 of its last 39 triggers, so `main` is verified more often than the 200-run window suggests. That window spans 18 hours and the push and pull request lanes fill it. The two triggers that published nothing did not fail and were not cancelled by `cancel-in-progress`, which resolves to `false` for `schedule`. They executed ZERO jobs: runs `32140419182` and `32206456661` both return `startedAt: null` for every job and were cancelled the second their successor was created, 2 out of 2. The mechanism is the other half of GitHub's concurrency contract. A group holds one pending run, and a third arrival cancels it. Every scheduled run keys into `ci-schedule-refs/heads/main-mudler/vllm.cpp`, and the suite now runs long enough to reach queue depth two: run `32118587477` took 9 h 28 min with no predecessor to wait for, on 345 job-minutes of work, because a 3.8-minute `ubuntu-latest` job waited 3 h 59 min for a runner. The spec proposes one rule over all 11 concurrency blocks: a group that never cancels must vary per run, because a queue that discards its own contents costs a verdict and reclaims nothing. It records why widening the cron is left to the operator as a separate decision, and why the 17 completed failures are not one systemic red. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [Claude Code]
…he baseline (#274) `cancel-in-progress: false` is half of GitHub's concurrency contract. The other half is the queue, and it holds exactly ONE pending run: a third run joining a group that has one run in progress and one pending cancels the pending one. The baseline lane keyed every scheduled run into `ci-schedule-refs/heads/main-<repo>`, so non-cancellable never meant it could not be cancelled. It cost 2 of the last 39 scheduled baselines, and neither executed a job. Runs 32140419182 and 32206456661 return `startedAt: null` for every job and died at 16:46:51 and 04:49:54, the seconds their successors 32162114781 and 32217173498 were created, 2 out of 2. Queue depth two is reachable because the suite outgrew the cron: run 32118587477 took 9 h 28 min with no predecessor to wait for, on 345 job-minutes of work, because a 3.8-minute `ubuntu-latest` job waited 3 h 59 min for a runner. All 11 concurrency blocks now admit `github.run_id` on `schedule` and `workflow_dispatch` only. For `push` the conditional resolves to `github.ref` and for `pull_request` to the pull request number, so both keys resolve to exactly what they resolved to before and #822's latest-only push lane is untouched. Widening the cron is left as a separate operator decision, with the reason in the spec: it is probabilistic, it spends the ~9-commit red range the 4h cadence bought, and the contention it answers is one day old. `GroupEvictionTests` holds the invariant over every concurrency block the file declares, derived from `cancel-in-progress` rather than from a list, so a job that joins later is covered by the pull request that adds it. Both directions are asserted, because a key that varies per run cancels nothing. Red before on 82 subtests across 5 tests at 250db75, green after at 65 tests rc=0. The two rewritten tests each keep their own direction and add the opposite one; `main-verifiability.md` gains the `## Outcome` it owed, which records that its risk 2 was handled for one half of the contract only. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-1m [Claude Code]
Bring the branch up to date before landing the baseline-lane fix. FOLLOWING_AGENTS_PROTOCOL Issue: #274 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
localai-bot
pushed a commit
that referenced
this pull request
Aug 19, 2026
Brings in the CI concurrency-group fix (#1323) and the DFlash2 scoping spec (#1321). Neither touches the decode-graph seam, `vt::PersistentStepInput` or the async decline this branch repairs. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cancel-in-progress: falseis half of GitHub's concurrency contract. The otherhalf is the queue, and it holds exactly ONE pending run: a third run joining a
group that has one run in progress and one pending cancels the pending one. The
baseline lane keyed every scheduled run into
ci-schedule-refs/heads/main-<repo>, so non-cancellable never meant it couldnot be cancelled.
.agents/specs/main-verifiability.mdnamed this as risk 2 andrecorded the handling as "own group plus
cancel-in-progress: false", whichcovers one half.
What the measurement actually says
gh run list --workflow ci.yml --limit 200reads 175 cancelled, 17 failure, 0success. That window spans 18 hours 14 minutes, because the push and pull
request lanes fill it: 135 cancelled and 15 failed pull request runs, 38
cancelled push runs, 5 scheduled runs. Read the lane instead, over its last 39
completed runs from
2026-08-12T17:10Zto2026-08-19T04:49Z: 20success, 17failure, 2cancelled. The lane published a verdict for 37 of 39 triggers,and the last fully green baseline is run
32067210005at76f2a6d84e41.So
mainis verified more often than the window suggests, and the defect is the2, not the 175.
The eviction, and it is not
cancel-in-progressBoth cancelled scheduled runs executed ZERO jobs.
3214041918213:05:5616:46:5132162114781at16:46:503220645666101:51:4804:49:5432217173498at04:49:53gh run view <id> --json jobsreturnsstartedAt: nullfor every job of both.The cancel instant equals the successor's creation instant, 2 out of 2, and the
queue is head-to-tail:
32162114781started its first job at18:20:21, thesecond
32118587477completed at18:20:20.Queue depth two is reachable because the suite outgrew the cron. Run
32118587477took 9 h 28 min with no predecessor to wait for, on 345job-minutes of work:
agent-recordis a 3.8-minuteubuntu-latestjob andwaited 3 h 59 min for a runner,
baseline-summaryis a 0.2-minute job andwaited 4 h 33 min. 8 of 39 runs ran over four hours.
The change
All 11 concurrency blocks admit
github.run_idonscheduleandworkflow_dispatchonly. Forpushthe conditional resolves togithub.refand for
pull_requestto the pull request number, so both keys resolve toexactly what they resolved to before and #822's latest-only push lane is
untouched. A queue that discards its own contents costs a verdict and reclaims
nothing, because a discarded run never ran anything to reclaim.
The cron stays at
17 */4 * * *. Widening it was rejected as the fix and leftas a separate operator decision: it is probabilistic, because the wall is set by
pool contention this file does not control; it spends the ~9-commit red range
the 4h cadence was chosen for, since 12h makes it ~27; and the contention it
answers is one day old, as runs from
2026-08-12to2026-08-17took 98 to 130minutes. Flipping
cancel-in-progressforpushstays rejected on the samearithmetic as #274, restated at the current suite cost: 55 pushes/day at 345
job-minutes is about 316 hours of job time per day.
Cost. The lane's own cost rises by the runs no longer discarded, 2 in 39 or
about 5 percent. The push and pull request lanes cost exactly what they cost
today. Peak baseline concurrency rises from 1 to
ceil(wall / 4 h), which is 3at the worst wall observed; total demand does not rise with it, because
serialization deferred the work rather than removing it.
The gate
GroupEvictionTestsholds the invariant over every concurrency block the filedeclares, enumerated from the parsed workflow rather than from a list, so a job
that joins later is covered by the pull request that adds it. It is derived from
cancel-in-progressand asserts both directions, because a key that varies perrun cancels nothing:
The resolver is new and value-valued.
resolve_booleanalready answered "doesthis expression mean true for this event"; a group key is not a boolean and is
no longer a constant, so half-resolving the string cannot see a conditional.
resolve_groupevaluates the same grammar over values with GitHub's truthiness,and
varies_per_runresolves one key against two synthetic runs. That turns"does this key carry
github.run_id" into the property that matters: do tworuns land in the same group.
Two existing tests are rewritten, each keeping its own direction and adding the
opposite one.
test_each_resolved_group_is_the_base_key_plus_the_event_constantnow resolves the whole key instead of only
${{ github.event_name }}, and keepsits equality against the base revision's key for
pushandpull_request.test_no_group_key_carries_a_run_varying_tokenblocked the token by substringfor every event, which was right about the contributor lanes and wrong about the
baseline lane; it becomes
test_a_baseline_group_is_unique_per_run_and_a_contributor_group_is_notandasserts the resolved property in both directions. Neither removes an assertion.
Red before, at
250db75a2: 82 subtest failures across 5 tests, 65 tests ran.Green after: 65 tests, rc 0.
Six mutations, each applied to the fixed tree, each verified as applied by its
own replacement count and
git diff --numstat, each restored and checkedbyte-for-byte by sha256:
${{ github.sha }}added to one job keytest_the_workflow_level_group_is_unique_per_baseline_runcancel-in-progresspolarity inverted on one jobtest_cancellation_resolves_to_the_right_boolean_for_every_eventconcurrency:block deletedtest_the_enumeration_finds_every_block_this_suite_knows_aboutThe fourth is the one that matters for scope: a fix applied only to the job
blocks leaves the suite green and the lane broken in exactly the measured way,
because both discarded runs were discarded as runs, before any job existed.
A seventh attempt never matched its anchor. The harness asserted on the
replacement count and refused rather than reporting the
OKit would otherwisehave printed, so it is recorded as no result rather than as a pass.
What this does not fix, and why the lane stays red
The lane reports RED, and that is the lane working.
windows-msvc-cpuandwindows-msvc-vulkanfail in 5 of the 5 scheduled runs since #503 put them onthe lane on 2026-08-17, both with
test_openai_api_server.exe exited with status -1073740791, which is0xC0000409,STATUS_STACK_BUFFER_OVERRUN. That is #584, open andpre-existing, and
ci.yml's own comment predicted it before the lane could seethat job at all.
The 17 completed failures are not one systemic red.
build-newest-gccfailed 2of 5 on
'::getpid' has not been declaredunder gcc 16, whicheb770f595already repaired for #1296 and current
maindoes not carry;sanitize-cpuandbuild-test-cpufailed 1 of 5;agent-recordfailed 2 of 39. Thedocumentation-checkpointandcommit-protocol-tagfailures seen on pullrequests are per-commit gates the baseline lane does not run.
Records
.agents/specs/main-verifiability.mdgains the## Outcomeit owed. All six ofits post-merge observations are now made, and it records that its risk 2 was
handled for one half of the contract only. #1316 is filed for a finding read out
of the two discarded runs:
scripts/main-baseline.pyrenders a run thatexecuted zero jobs as
REDwith all 11 jobsmissing, soNEWEST BASELINE: RED at <sha>names a tree the run never checked out. It isfail-closed and this change removes its only observed producer, so it is owed
under
## Owedof.agents/specs/baseline-lane-eviction.mdrather than foldedin: separating "narrowed" from "never started" changes what the verdict means.
What the pre-merge evidence cannot establish
This pull request's own run is a
pull_requestevent, so it exercises neitherthe
scheduletrigger nor the main-branch grouping, and GitHub firesscheduleonly for the workflow committed on the default branch. Everything green here is
static. The claim only a post-merge observation settles: a scheduled run that
starts while another is in progress reaches a completed verdict rather than
cancelledwith zero jobs. That condition occurred 8 times in the 39-runwindow, so it should be visible within a day.
Tracked by #274.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5-1m [Claude Code]