Skip to content

ci: a group that never cancels must never queue, or GitHub discards the baseline (#274) - #1323

Merged
localai-bot merged 3 commits into
mainfrom
fix/baseline-lane-eviction
Aug 19, 2026
Merged

ci: a group that never cancels must never queue, or GitHub discards the baseline (#274)#1323
localai-bot merged 3 commits into
mainfrom
fix/baseline-lane-eviction

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

cancel-in-progress: false is half of GitHub's concurrency contract. The other
half is the queue, and it holds exactly ONE pending run: a third run joining a
group that has one run in progress and one pending cancels the pending one. The
baseline lane keyed every scheduled run into
ci-schedule-refs/heads/main-<repo>, so non-cancellable never meant it could
not be cancelled. .agents/specs/main-verifiability.md named this as risk 2 and
recorded the handling as "own group plus cancel-in-progress: false", which
covers one half.

What the measurement actually says

gh run list --workflow ci.yml --limit 200 reads 175 cancelled, 17 failure, 0
success. That window spans 18 hours 14 minutes, because the push and pull
request lanes fill it: 135 cancelled and 15 failed pull request runs, 38
cancelled push runs, 5 scheduled runs. Read the lane instead, over its last 39
completed runs from 2026-08-12T17:10Z to 2026-08-19T04:49Z: 20 success, 17
failure, 2 cancelled. The lane published a verdict for 37 of 39 triggers,
and the last fully green baseline is run 32067210005 at 76f2a6d84e41.

So main is verified more often than the window suggests, and the defect is the
2, not the 175.

The eviction, and it is not cancel-in-progress

Both cancelled scheduled runs executed ZERO jobs.

Run Created Jobs started Cancelled at Successor created
32140419182 13:05:56 none 16:46:51 32162114781 at 16:46:50
32206456661 01:51:48 none 04:49:54 32217173498 at 04:49:53

gh run view <id> --json jobs returns startedAt: null for every job of both.
The cancel instant equals the successor's creation instant, 2 out of 2, and the
queue is head-to-tail: 32162114781 started its first job at 18:20:21, the
second 32118587477 completed at 18:20:20.

Queue depth two is reachable because the suite outgrew the cron. Run
32118587477 took 9 h 28 min with no predecessor to wait for, on 345
job-minutes of work: agent-record is a 3.8-minute ubuntu-latest job and
waited 3 h 59 min for a runner, baseline-summary is a 0.2-minute job and
waited 4 h 33 min. 8 of 39 runs ran over four hours.

The change

All 11 concurrency blocks admit github.run_id on schedule and
workflow_dispatch only. For push the conditional resolves to github.ref
and for pull_request to the pull request number, so both keys resolve to
exactly what they resolved to before and #822's latest-only push lane is
untouched. A queue that discards its own contents costs a verdict and reclaims
nothing, because a discarded run never ran anything to reclaim.

The cron stays at 17 */4 * * *. Widening it was rejected as the fix and left
as a separate operator decision: it is probabilistic, because the wall is set by
pool contention this file does not control; it spends the ~9-commit red range
the 4h cadence was chosen for, since 12h makes it ~27; and the contention it
answers is one day old, as runs from 2026-08-12 to 2026-08-17 took 98 to 130
minutes. Flipping cancel-in-progress for push stays rejected on the same
arithmetic as #274, restated at the current suite cost: 55 pushes/day at 345
job-minutes is about 316 hours of job time per day.

Cost. The lane's own cost rises by the runs no longer discarded, 2 in 39 or
about 5 percent. The push and pull request lanes cost exactly what they cost
today. Peak baseline concurrency rises from 1 to ceil(wall / 4 h), which is 3
at the worst wall observed; total demand does not rise with it, because
serialization deferred the work rather than removing it.

The gate

GroupEvictionTests holds the invariant over every concurrency block the file
declares, enumerated from the parsed workflow rather than from a list, so a job
that joins later is covered by the pull request that adds it. It is derived from
cancel-in-progress and asserts both directions, because a key that varies per
run cancels nothing:

never cancels on an event  =>  the group MUST vary per run
cancels on an event        =>  the group MUST NOT vary per run

The resolver is new and value-valued. resolve_boolean already answered "does
this expression mean true for this event"; a group key is not a boolean and is
no longer a constant, so half-resolving the string cannot see a conditional.
resolve_group evaluates the same grammar over values with GitHub's truthiness,
and varies_per_run resolves one key against two synthetic runs. That turns
"does this key carry github.run_id" into the property that matters: do two
runs land in the same group.

Two existing tests are rewritten, each keeping its own direction and adding the
opposite one. test_each_resolved_group_is_the_base_key_plus_the_event_constant
now resolves the whole key instead of only ${{ github.event_name }}, and keeps
its equality against the base revision's key for push and pull_request.
test_no_group_key_carries_a_run_varying_token blocked the token by substring
for every event, which was right about the contributor lanes and wrong about the
baseline lane; it becomes
test_a_baseline_group_is_unique_per_run_and_a_contributor_group_is_not and
asserts the resolved property in both directions. Neither removes an assertion.

Red before, at 250db75a2: 82 subtest failures across 5 tests, 65 tests ran.
Green after: 65 tests, rc 0.

Six mutations, each applied to the fixed tree, each verified as applied by its
own replacement count and git diff --numstat, each restored and checked
byte-for-byte by sha256:

Mutation Applied to Result
${{ github.sha }} added to one job key 1 line 8 failures, 4 tests, including the base-key equality it was originally built for
run id admitted unconditionally in every key 11 lines 82 failures, 5 tests
one job key reverted to the shared shape 1 line 8 failures, 4 tests
only the workflow-level key reverted 1 line 6 failures, 3 tests, including test_the_workflow_level_group_is_unique_per_baseline_run
cancel-in-progress polarity inverted on one job 1 line 8 failures, including the pre-existing test_cancellation_resolves_to_the_right_boolean_for_every_event
a whole concurrency: block deleted 3 lines 5 failures, including test_the_enumeration_finds_every_block_this_suite_knows_about

The fourth is the one that matters for scope: a fix applied only to the job
blocks leaves the suite green and the lane broken in exactly the measured way,
because both discarded runs were discarded as runs, before any job existed.

A seventh attempt never matched its anchor. The harness asserted on the
replacement count and refused rather than reporting the OK it would otherwise
have printed, so it is recorded as no result rather than as a pass.

What this does not fix, and why the lane stays red

The lane reports RED, and that is the lane working. windows-msvc-cpu and
windows-msvc-vulkan fail in 5 of the 5 scheduled runs since #503 put them on
the lane on 2026-08-17, both with
test_openai_api_server.exe exited with status -1073740791, which is
0xC0000409, STATUS_STACK_BUFFER_OVERRUN. That is #584, open and
pre-existing, and ci.yml's own comment predicted it before the lane could see
that job at all.

The 17 completed failures are not one systemic red. build-newest-gcc failed 2
of 5 on '::getpid' has not been declared under gcc 16, which eb770f595
already repaired for #1296 and current main does not carry; sanitize-cpu and
build-test-cpu failed 1 of 5; agent-record failed 2 of 39. The
documentation-checkpoint and commit-protocol-tag failures seen on pull
requests are per-commit gates the baseline lane does not run.

Records

.agents/specs/main-verifiability.md gains the ## Outcome it owed. All six of
its post-merge observations are now made, and it records that its risk 2 was
handled for one half of the contract only. #1316 is filed for a finding read out
of the two discarded runs: scripts/main-baseline.py renders a run that
executed zero jobs as RED with all 11 jobs missing, so
NEWEST BASELINE: RED at <sha> names a tree the run never checked out. It is
fail-closed and this change removes its only observed producer, so it is owed
under ## Owed of .agents/specs/baseline-lane-eviction.md rather than folded
in: separating "narrowed" from "never started" changes what the verdict means.

What the pre-merge evidence cannot establish

This pull request's own run is a pull_request event, so it exercises neither
the schedule trigger nor the main-branch grouping, and GitHub fires schedule
only for the workflow committed on the default branch. Everything green here is
static. The claim only a post-merge observation settles: a scheduled run that
starts while another is in progress reaches a completed verdict rather than
cancelled with zero jobs. That condition occurred 8 times in the 39-run
window, so it should be visible within a day.

Tracked by #274.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5-1m [Claude Code]

mudler added 3 commits August 19, 2026 07:32
…ict (#274)

The scheduled lane publishes a verdict for 37 of its last 39 triggers, so `main`
is verified more often than the 200-run window suggests. That window spans 18
hours and the push and pull request lanes fill it. The two triggers that
published nothing did not fail and were not cancelled by `cancel-in-progress`,
which resolves to `false` for `schedule`. They executed ZERO jobs: runs
`32140419182` and `32206456661` both return `startedAt: null` for every job and
were cancelled the second their successor was created, 2 out of 2.

The mechanism is the other half of GitHub's concurrency contract. A group holds
one pending run, and a third arrival cancels it. Every scheduled run keys into
`ci-schedule-refs/heads/main-mudler/vllm.cpp`, and the suite now runs long
enough to reach queue depth two: run `32118587477` took 9 h 28 min with no
predecessor to wait for, on 345 job-minutes of work, because a 3.8-minute
`ubuntu-latest` job waited 3 h 59 min for a runner.

The spec proposes one rule over all 11 concurrency blocks: a group that never
cancels must vary per run, because a queue that discards its own contents costs
a verdict and reclaims nothing. It records why widening the cron is left to the
operator as a separate decision, and why the 17 completed failures are not one
systemic red.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5-1m [Claude Code]
…he baseline (#274)

`cancel-in-progress: false` is half of GitHub's concurrency contract. The other
half is the queue, and it holds exactly ONE pending run: a third run joining a
group that has one run in progress and one pending cancels the pending one. The
baseline lane keyed every scheduled run into
`ci-schedule-refs/heads/main-<repo>`, so non-cancellable never meant it could
not be cancelled.

It cost 2 of the last 39 scheduled baselines, and neither executed a job. Runs
32140419182 and 32206456661 return `startedAt: null` for every job and died at
16:46:51 and 04:49:54, the seconds their successors 32162114781 and 32217173498
were created, 2 out of 2. Queue depth two is reachable because the suite outgrew
the cron: run 32118587477 took 9 h 28 min with no predecessor to wait for, on
345 job-minutes of work, because a 3.8-minute `ubuntu-latest` job waited 3 h 59
min for a runner.

All 11 concurrency blocks now admit `github.run_id` on `schedule` and
`workflow_dispatch` only. For `push` the conditional resolves to `github.ref`
and for `pull_request` to the pull request number, so both keys resolve to
exactly what they resolved to before and #822's latest-only push lane is
untouched. Widening the cron is left as a separate operator decision, with the
reason in the spec: it is probabilistic, it spends the ~9-commit red range the
4h cadence bought, and the contention it answers is one day old.

`GroupEvictionTests` holds the invariant over every concurrency block the file
declares, derived from `cancel-in-progress` rather than from a list, so a job
that joins later is covered by the pull request that adds it. Both directions
are asserted, because a key that varies per run cancels nothing.

Red before on 82 subtests across 5 tests at 250db75, green after at 65 tests
rc=0. The two rewritten tests each keep their own direction and add the opposite
one; `main-verifiability.md` gains the `## Outcome` it owed, which records that
its risk 2 was handled for one half of the contract only.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5-1m [Claude Code]
Bring the branch up to date before landing the baseline-lane fix.

FOLLOWING_AGENTS_PROTOCOL

Issue: #274

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]
@localai-bot
localai-bot merged commit f06b9e9 into main Aug 19, 2026
1 check failed
localai-bot pushed a commit that referenced this pull request Aug 19, 2026
Brings in the CI concurrency-group fix (#1323) and the DFlash2 scoping spec
(#1321). Neither touches the decode-graph seam, `vt::PersistentStepInput` or the
async decline this branch repairs.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [Claude Code]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants