Add frozen DS4 W2 quality and context harness - #7
Open
OmarB97 wants to merge 1 commit into
Open
Conversation
This was referenced Jul 12, 2026
9prodhi
added a commit
to 9prodhi/vLLM-Moet
that referenced
this pull request
Jul 13, 2026
…sized-resume truncate, chmod fsync, no monitor/fd leak Re-review wave 2 (5 items on tools/plane_file.py, tools/build_plane_file.py, tools/test_plane_file.py): - [kacper-daftcode#7] CLI verify/build catch ALL exceptions at the boundary (KeyError/ TypeError from malformed-but-checksum-valid metadata, OSError) and exit non-zero with a one-line clean error; tracebacks behind a new --debug flag. - [kacper-daftcode#8] RSS budget is a HARD gate: _enforce_budget() aborts the build BEFORE finalization when the sampled monitor observes the budget exceeded; the .tmp is kept for resume and no finalized artifact is produced. - [kacper-daftcode#10] finalize() fsyncs the file inode after os.chmod(0o444) so the mode change is durable, not just the rename. - [new Important] resume normalizes an OVERSIZED .tmp: ftruncate back to the exact layout length (was: grow-only), never finalize a file verify rejects. - [new Minor] mon.start() + PlaneFileWriter construction moved inside the try/finally, monitor thread named + stop() guarded, and the writer ctor closes its fd if post-open validation raises: a resume-validation failure leaks neither the monitor thread nor an fd. TDD (failing test first) for kacper-daftcode#7, kacper-daftcode#8, and the oversized-resume item; regression tests for the leak item. Full suite: 73 passed, 0 failed (golden on CUDA). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
W2 quality claims need a frozen scorer, deterministic prompt set, explicit warmup/pool gates, and exact context-token admission. Ad hoc transcripts cannot distinguish a real sink from formatting-only mismatches or calibration attempts.
What changed
How to review
Read
tools/ds4_eval/harness.pyfor frozen scoring and admission rules, thencontext_probe.pyfor tokenizer-only calibration versus inference. Review the tests alongside each module and confirm rejected calibration candidates cannot count as chat attempts.Evidence
The harness produced the recorded comprehensive 120/120 semantic, zero-sink three-seed result and 3/3 exact 120K context series. The frozen-head supplement produced 80/80 semantic, zero sinks, 20/20 retry-free prewarms, four passing pool gates, and one exact deepest 120K request.
Verification
python3 -m pytest tools/ds4_eval/tests/test_context_probe.py tools/ds4_eval/tests/test_harness.py -q— 33 passed, 17 subtests passedruff check tools/ds4_evalruff format --check tools/ds4_evalpython3 -m py_compile tools/ds4_eval/*.py tools/ds4_eval/tests/test_context_probe.py tools/ds4_eval/tests/test_harness.pygit diff --checkRisks / gaps
The 40-item set is a frozen regression instrument, not a general benchmark. Live controller guard tests remain private because they embed deployment topology; the public watchdog contract is covered in the separate safety-probes PR.
Collaborators