Skip to content

fix(planner): FP16-baseline consistency for quality/tier (#4) + cut 0.6.2 - #10

Merged
Sahil170595 merged 1 commit into
mainfrom
fix/0.6.2-quality-consistency
Jul 26, 2026
Merged

fix(planner): FP16-baseline consistency for quality/tier (#4) + cut 0.6.2#10
Sahil170595 merged 1 commit into
mainfrom
fix/0.6.2-quality-consistency

Conversation

@Sahil170595

Copy link
Copy Markdown
Owner

Summary

Completes the fix for #4. 0.6.1 stopped quality_tier returning unknown for llama3.1-8b, but only patched quality_tier()estimate() and predict() stayed on the old FP16-baseline chain. Result: the FP16 config reported quality 0.5 with provenance unknown (ranking the highest-precision option below Q2_K at 0.59), while its tier said negligible. Quality and tier disagreed.

This unifies all three methods behind one _fp16_baseline resolver, then cuts 0.6.2.

The fix

QualityModel.predict(), estimate(), and quality_tier() now share _fp16_baseline(model, family) — one resolution chain:

  1. measured FP16 baseline (fp16_baselines)
  2. explicit model|FP16 lookup entry
  3. the model's own highest-precision measured quant (TR125 uses Q8_0 as the llama3.1-8b baseline — 8B FP16 = 16 GB doesn't fit the RTX 4080, so no FP16 row exists)
  4. family mean (off-registry models)

Quality, provenance, and tier can no longer diverge. No FP16 number is fabricated in fitted_models.json — the anchor is the honest one TR125 already uses.

Before → after (plan --model-size 8b --hardware "RTX 4090 24GB" --request-rate 2.0 --json)

quant before after
FP16 quality=0.5, prov=unknown, tier=negligible (inconsistent) quality=0.635, prov=estimated, tier=negligible
Q8_0/Q6_K/… unchanged (measured) unchanged (measured)

Tests

  • New regression test_estimate_and_tier_share_baseline: estimate() and quality_tier() resolve the same baseline; FP16 is estimated (not the 0.5/unknown prior) and not ranked below Q2_K.
  • Full suite: 488 passed, ruff clean.

Release

  • 0.6.10.6.2 (pyproject.toml, __init__.py)
  • CHANGELOG.md [0.6.2] entry
  • CLAUDE.md version + test-count refresh (483 → 488, correcting pre-existing drift)

…r diverge (#4)

0.6.1 fixed quality_tier() for llama3.1-8b but left estimate()/predict() on the
old baseline chain. FP16 therefore reported quality 0.5 / provenance "unknown" --
ranking the highest-precision config below Q2_K (0.59) -- while its tier said
"negligible". Extract one _fp16_baseline resolver (measured FP16 -> the model's
highest measured quant -> family mean) shared by predict/estimate/quality_tier,
so the quality float, its provenance, and the tier are always computed off the
same baseline. FP16 now reports 0.635 estimated, consistent with negligible.

Bump 0.6.1 -> 0.6.2; add regression test; refresh CHANGELOG and CLAUDE.md counts.
@Sahil170595
Sahil170595 merged commit a7b36e7 into main Jul 26, 2026
6 checks passed
@Sahil170595
Sahil170595 deleted the fix/0.6.2-quality-consistency branch July 26, 2026 02:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant