fix(planner): FP16-baseline consistency for quality/tier (#4) + cut 0.6.2 - #10
Merged
Merged
Conversation
…r diverge (#4) 0.6.1 fixed quality_tier() for llama3.1-8b but left estimate()/predict() on the old baseline chain. FP16 therefore reported quality 0.5 / provenance "unknown" -- ranking the highest-precision config below Q2_K (0.59) -- while its tier said "negligible". Extract one _fp16_baseline resolver (measured FP16 -> the model's highest measured quant -> family mean) shared by predict/estimate/quality_tier, so the quality float, its provenance, and the tier are always computed off the same baseline. FP16 now reports 0.635 estimated, consistent with negligible. Bump 0.6.1 -> 0.6.2; add regression test; refresh CHANGELOG and CLAUDE.md counts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Completes the fix for #4. 0.6.1 stopped
quality_tierreturningunknownforllama3.1-8b, but only patchedquality_tier()—estimate()andpredict()stayed on the old FP16-baseline chain. Result: the FP16 config reported quality0.5with provenanceunknown(ranking the highest-precision option belowQ2_Kat0.59), while its tier saidnegligible. Quality and tier disagreed.This unifies all three methods behind one
_fp16_baselineresolver, then cuts 0.6.2.The fix
QualityModel.predict(),estimate(), andquality_tier()now share_fp16_baseline(model, family)— one resolution chain:fp16_baselines)model|FP16lookup entryQ8_0as thellama3.1-8bbaseline — 8B FP16 = 16 GB doesn't fit the RTX 4080, so no FP16 row exists)Quality, provenance, and tier can no longer diverge. No FP16 number is fabricated in
fitted_models.json— the anchor is the honest one TR125 already uses.Before → after (
plan --model-size 8b --hardware "RTX 4090 24GB" --request-rate 2.0 --json)quality=0.5,prov=unknown,tier=negligible(inconsistent)quality=0.635,prov=estimated,tier=negligibleTests
test_estimate_and_tier_share_baseline:estimate()andquality_tier()resolve the same baseline; FP16 isestimated(not the0.5/unknownprior) and not ranked below Q2_K.Release
0.6.1→0.6.2(pyproject.toml,__init__.py)CHANGELOG.md[0.6.2]entryCLAUDE.mdversion + test-count refresh (483 → 488, correcting pre-existing drift)