Skip to content

fix(ci): the residue after #190 — ledger anchor, env var, dead tests, #154 docs - #188

Merged
mudler merged 1 commit into
mainfrom
row/ENG-CI-GREEN
Aug 9, 2026
Merged

fix(ci): the residue after #190 — ledger anchor, env var, dead tests, #154 docs#188
mudler merged 1 commit into
mainfrom
row/ENG-CI-GREEN

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

Why

main has been red on every run overnight. I surveyed the last eight runs and sorted the failures by scope, because that distinction is what actually matters here:

  • Diff-scoped gates (POL-PR-REQUIRED, POL-COMMIT-TRAILERS, POL-DOC-*) fail over one push range and self-heal on the next correct landing.
  • Tree-scoped gates fail on every run and every open PR until someone fixes them. Four were red, which is why unrelated contributor PRs were blocked.

This fixes all four, plus the docs #154 owed.

The four tree-scoped failures

1. check-public-doc-tablesdocs/STATUS.md 277214 chars over a 276960 shrink-only ratchet.

The DeepSeek last-mile Bricks 3 and 4 were a run-by-run log of a NOT-pushed branch — 3871 chars for two negative results — on a page whose contract is one binding line per capability. Collapsed to 1009 chars that keep both negatives and the finding they establish: dp4a near-flat +0.5%, aligned repack a MEASURED NEGATIVE at +4.7 GiB peak RSS, coalesced-load hypothesis REFUTED, the Q8_0 matvec LATENCY/OCCUPANCY-bound at ~61% of ds4. The nsys tables were already in .agents/benchmark-record.md.

Ratchet lowered 276960 → 274760. The 2026-08-08 ratchet left ~15 chars of headroom — that one was mine and it was too tight: three Vulkan landings that night each added a status line and pushed the page over within hours, which is how this became a blocker. New headroom is 408, so the ratchet measures bloat rather than merge timing. Still a large net shrink, the only direction it may move.

2. check-env-doc — 12 undocumented production env vars.

Seven VT_GEMMA4_*, three VT_ROCM_*, and the two VT_SERVER_MAX_* request limits. Documented in docs/ENVIRONMENT.md beside their siblings rather than allowlisted, because the existing VT_GEMMA4_RESIDENT_* and VT_ROCM_ATTN_CPU_REF knobs live there. Every default was read from the source, not guessed.

3. check-state-order — the 2026-08-09T13:00 anchor sat before its heading instead of on the line after it, so that entry counted as unanchored.

4. check-now-currentNOW.md 6080 chars against a 6000 budget. Dropped the two rows carrying no open work: MXFP4 parity (marked TERMINAL) and the supported-models list (LANDED, residual -). Both keep their evidence in docs/FEATURES.md and the matrices.

The docs #154 owed

#154 is what reddened documentation-checkpoint when it landed. It shipped eight real user-facing flags with no USAGE.md entry: --repeat on vllm-cli, and --enable/--disable-log-requests, --enable-log-outputs, --max-log-len, --enable/--disable-metrics, --enable/--no-enable-thinking, --verbose on the server. Defaults read from examples/server/main.cpp.

Not documented: --default-chat-template-kwargs. It appears only in a source comment naming vLLM's spelling and is not a parsed flag — documenting it would have invented a feature.

docs/BENCHMARKS.md: the reproduce table still said the Vulkan-vs-llama.cpp arm was "Not yet runnable (no model runs on Vulkan)". It runs; that row now names the actual harness.

Two things I deliberately did not do

README is untouched, and it is still wrong. It says "24 native ops" in two places where STATUS and NOW both say 25 — .agents/state.md already flagged this. POL-DOC-README only accepts a README edit alongside one of six landing-source files (.agents/mission.md, CMakeLists.txt, benchmarks/demo/*.json, examples/{cli,server}/main.cpp), and a stale-number correction touches none of them and arguably should not have to. This needs a ruling: waive it, or let the gate accept a correction whose benchmark source already sits in docs/BENCHMARKS.md:365. I did not want to manufacture an unrelated landing-source edit to slip a factual fix past a gate.

POL-PR-REQUIRED is not "fixed" because it is not broken. It keeps firing on content commits from --merge landings and direct pushes to main — the gate is working as designed. The permanent fix is the repository setting (allow squash only, disable merge commits), which needs admin.

Gates

All 20 tree-scoped checkers green, plus test_agent_record, test_doc_checkpoint, test_check_public_doc_tables, test_check_readme_structure, test_check_env_doc, test_check_state_order, test_check_now_current, test_agent_role, test_agent_gates, test_check_gate_commands, test_audit_live_rows — on a worktree pinned at f921062b. check-commit-trailers and check-doc-checkpoint pass over the range. No code touched.

Please squash-merge.

@mudler
mudler force-pushed the row/ENG-CI-GREEN branch 2 times, most recently from 446e2d2 to b8a5309 Compare August 9, 2026 07:55
@localai-bot localai-bot changed the title fix(ci): restore main's four red tree-scoped gates, document #154's flags fix(ci): restore main's red tree-scoped gates + two inert/broken checkers Aug 9, 2026
@localai-bot

Copy link
Copy Markdown
Collaborator Author

Pushed two more fixes after pr-size failed on this PR — both are real defects the survey surfaced, not paperwork.

5. tests/scripts/test_check_public_doc_tables.py was half dead. Its if __name__ == "__main__": unittest.main() sat at line 362, before the StatusRatchet class at line 392. CI runs this file as a script, so unittest.main() executed before that class existed and all 8 of its tests — including test_growth_past_the_char_ratchet_is_rejected and test_the_live_page_is_inside_its_ratchetnever ran. The ratchet's own mutation suite was inert. Moving the block to the end takes the file from 41 to 49 collected tests; all 49 pass.

I only found this because check-pr-size demands mutation evidence for a checker change, and the new test I added didn't fail when it should have. It wasn't failing because it wasn't running.

The new test pins what this PR actually changes — headroom is bounded, so a red char ratchet can't be cleared by inflating the number instead of shrinking the page. Two-way proof (bytecode disabled):

mutation result
chars=279000 FAILED — 4648 not less than or equal to 2000
chars=274000 FAILED — -352 not greater than or equal to 0
baseline 274760 OK, 49 tests

6. check-pr-size classified neither CLAUDE.md nor MANIFESTO.md. classify_path raised on both, so any PR touching either failed the gate outright. CLAUDE.md arrived via the b8d996e5 direct push and MANIFESTO.md via a9a8581d, neither registering a path class. Both are top-level reader-facing prose — same class as README. Mutation: dropping CLAUDE.md back out gives Lists differ: ['CLAUDE.md'] != [].

Note test_check_pr_size.py is not wired into ci.yml — it only runs via python3 -m unittest, which is why this sat unnoticed. Worth wiring up, but I've left that as a separate call since it may surface more.

@mudler
mudler force-pushed the row/ENG-CI-GREEN branch from b8a5309 to 3abaffe Compare August 9, 2026 09:03
…#154 docs

#190 landed while this branch was open and fixed more of main's red than
this did, and did it better: the STATUS.md ratchet cleared by MOVING a
33,211-char cell verbatim per POL-EVIDENCE-PRESERVE rather than condensing
it, CLAUDE.md/MANIFESTO.md classified (CLAUDE.md as procedure -- it is a
symlink to AGENTS.md -- which is more accurate than the public_document I
had used), a generated path class for SPIR-V, and a real
benchmarks/demo/vulkan_27b_llamacpp.json landing source so the false
README could finally be corrected. Everything this branch had that #190
also covers is dropped rather than re-litigated.

What was still red on main at c1716fd, both TREE-scoped, so both were
failing every run and every open PR:

1. check-agent-record: .agents/parity-ledger.md carried
   `../examples/server/main.cpp#L90`, and #189 turned that file into a
   23-line thin ABI client, so the anchor pointed past EOF. The referenced
   "server flag" is the CUDA-graph-replay profiler trigger, which now
   lives at src/vllm/entrypoints/openai/server_main.cpp:692
   (`args.cuda_profile_graph_replays > 0`). Repointed there. Only the
   pointer moves; the ledger entry is untouched.

2. check-env-doc: VLLM_CPP_HTTP_FIXED_POOL, added by #189, was
   undocumented. Documented in docs/ENVIRONMENT.md rather than
   allowlisted, because it changes runtime behavior (`=0` reverts the HTTP
   worker pool to legacy dynamic) and the comparable A/B switches
   VT_ROCM_ATTN_CPU_REF and VT_DEBUG_SAMPLED are documented there.

3. tests/scripts/test_check_public_doc_tables.py still has its
   `if __name__ == "__main__": unittest.main()` at line 362, BEFORE the
   StatusRatchet class at 392. CI runs this file as a script, so
   unittest.main() executes before that class is defined and all 8 of its
   tests -- including test_growth_past_the_char_ratchet_is_rejected and
   test_the_live_page_is_inside_its_ratchet -- never run. The ratchet's own
   mutation suite is inert, which is notable given #190 just re-pinned that
   ratchet. Moving the block to the end takes the file from 41 to 49
   collected tests; all 49 pass against #190's 244486.

4. docs/USAGE.md: #154 shipped eight user-facing flags with no USAGE entry
   and #189 did not add them either. Re-verified every one still parses
   after the thin-client refactor: --repeat in examples/cli/main.cpp, and
   --enable/--disable-log-requests, --enable-log-outputs, --max-log-len,
   --enable/--disable-metrics, --enable/--no-enable-thinking and --verbose
   in src/vllm/entrypoints/openai/server_main.cpp. NOT documented:
   --default-chat-template-kwargs, which appears only in a source comment
   naming vLLM's spelling and is not a parsed flag.

5. docs/BENCHMARKS.md still told the reader the Vulkan-vs-llama.cpp arm was
   "Not yet runnable (no model runs on Vulkan)". It runs; that row now
   names the harness.

6. docs/STATUS.md said the multimodal server seam "is wired into
   `examples/server/main.cpp`", which #189 made false -- the
   `preprocessor_config.json` guard now lives in
   src/vllm/entrypoints/openai/server_main.cpp (verified by grep, not
   assumed). Corrected. The longer path costs 19 chars against 3 of
   ratchet headroom, so it is paid for inside the same cell by tightening
   two clauses that restate what the sentence already says. Net -5.

Note for whoever touches docs/STATUS.md next: #190 tightened the char
ratchet to the measured byte, so slack is 3. Any status line added without
paying for it reds a tree-scoped gate again. That is a deliberate choice of
#190's, left alone here.

Gates: 14 tree-scoped checkers green, plus test_check_public_doc_tables
(49), test_agent_record, test_doc_checkpoint, test_check_state_order,
test_check_now_current, test_check_env_doc and test_check_pr_size (21), on
a worktree pinned at c1716fd. No product code touched.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude-Code:claude-opus-5 [Claude Code]
@mudler
mudler force-pushed the row/ENG-CI-GREEN branch from 3abaffe to a38e70b Compare August 9, 2026 09:03
@localai-bot localai-bot changed the title fix(ci): restore main's red tree-scoped gates + two inert/broken checkers fix(ci): the residue after #190 — ledger anchor, env var, dead tests, #154 docs Aug 9, 2026
@localai-bot

Copy link
Copy Markdown
Collaborator Author

Rewritten against the new main. #190 landed while this was open and fixed more of main's red than this did — and did it better: the STATUS.md ratchet cleared by moving a 33,211-char cell verbatim per POL-EVIDENCE-PRESERVE rather than condensing it, CLAUDE.md/MANIFESTO.md classified (CLAUDE.md as procedure — it's a symlink to AGENTS.md — which is more accurate than the public_document I'd used), a generated path class for SPIR-V, and a real benchmarks/demo/vulkan_27b_llamacpp.json landing source so the false README could finally be corrected.

Everything #190 also covers is dropped, not re-litigated. This branch is now rebased onto bd20da30 and reduced to what is still actually broken:

# still red / missing fix
1 check-agent-recordparity-ledger.md anchored examples/server/main.cpp#L90; #189 made that file 23 lines, so it pointed past EOF repointed to server_main.cpp:692, the flag's real home. Pointer only; the ledger entry is untouched
2 check-env-docVLLM_CPP_HTTP_FIXED_POOL (added by #189) undocumented documented in ENVIRONMENT.md; it changes runtime behaviour, and the comparable A/B switches live there
3 8 tests never run unittest.main() still sits at line 362, before the StatusRatchet class at 392, so CI's script invocation never reaches them. 41 → 50 collected
4 #154's eight flags still undocumented added to USAGE.md, each re-verified to still parse after #189's thin-client refactor
5 BENCHMARKS still said Vulkan was "Not yet runnable (no model runs on Vulkan)" row now names the harness
6 STATUS said the mm seam is wired into examples/server/main.cpp false after #189; the preprocessor_config.json guard is in server_main.cpp (grepped, not assumed)

Item 3 is worth a second look given #190 just re-pinned that ratchet: its own mutation suite has never executed, so test_growth_past_the_char_ratchet_is_rejected and test_the_live_page_is_inside_its_ratchet have been decorative.

Note on item 6: the longer path costs 19 chars against 3 of headroom (#190 tightened to the measured byte deliberately), so it's paid for inside the same cell by tightening two clauses that restate what the sentence already says. Net −5. Anyone adding a STATUS line next will hit the same wall — that's #190's deliberate choice, left alone here.

Not documented: --default-chat-template-kwargs, which appears only in a source comment naming vLLM's spelling and is not a parsed flag.

@mudler
mudler merged commit dc2139b into main Aug 9, 2026
10 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants