fix(planner): FP16-baseline consistency for quality/tier (#4) + cut 0.6.2

CHANGELOG.md

@@ -7,6 +7,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.6.2] - 2026-07-25
+
+### Fixed
+- **`quality_tier` and the quality float now agree for `llama3.1-8b` FP16**
+  (#4 follow-up). 0.6.1 fixed `quality_tier` but left `estimate()` / `predict()`
+  on the old baseline chain, so the FP16 config reported quality `0.5` with
+  provenance `unknown` -- ranking the *highest-precision* option below `Q2_K`
+  (`0.59`) -- while its tier said `negligible`. All three methods now share one
+  `_fp16_baseline` resolver (measured FP16 -> the model's highest measured quant
+  -> family mean), so quality, provenance, and tier can never diverge. FP16 now
+  reports `0.635` (estimated), consistent with its `negligible` tier.
+
 ## [0.6.1] - 2026-07-04
 
 ### Fixed

CLAUDE.md

@@ -4,7 +4,7 @@
 
 ChimeraForge is an LLM inference benchmarking and deployment planning platform, broken out from the Banterhearts program. It provides quantified, reproducible answers to LLM deployment decisions, backed by ~204,000 real measurements on consumer GPUs. Ships both research artifacts (32 technical reports, TR108-TR137 + TR142/TR146) and production CLI tools (`chimeraforge plan` and `chimeraforge bench`).
 
-**Version:** 0.6.1 | **License:** MIT | **Python:** >=3.10 | **Rust:** >=1.70
+**Version:** 0.6.2 | **License:** MIT | **Python:** >=3.10 | **Rust:** >=1.70
 
 ## Quick Reference
 
@@ -36,7 +36,7 @@ chimeraforge plan --model qwen3:14b --measure   # bench live first, then plan (p
 # Run benchmarks (requires live Ollama)
 chimeraforge bench --model llama3.2-3b --runs 5
 
-# Run tests (483 total; 0.6.0 adds KV-batch/prefill-decode/continuous-batching/variance/pareto/accuracy + blind-audit regressions)
+# Run tests (488 total; 0.6.0 adds KV-batch/prefill-decode/continuous-batching/variance/pareto/accuracy + blind-audit regressions)
 pytest tests/ -v
 
 # Lint
@@ -53,7 +53,7 @@ cd src/rust/demo_multiagent && cargo build --release
 ```
 src/
   chimeraforge/                       # CLI tool + capacity planner (pip-installable)
-    __init__.py                       # Exports __version__ = "0.6.1"
+    __init__.py                       # Exports __version__ = "0.6.2"
     cli.py                            # Typer entry point, registers plan/suggest/safety/... (lazy imports)
     commands/                         # One module per CLI command (plan.py, suggest.py, safety.py, ...)
     planner/
@@ -119,7 +119,7 @@ experiments/                          # TR108-TR133 experiment folders
 data/                                 # baselines/, csv/, research/
 outputs/publish_ready/                # Final reports and notebooks
 scripts/                              # Mostly scaffolded (empty); setup_ollama_model.ps1 is live
-tests/                                # 19 files, 483 tests (planner/bench split per-concern; test_accuracy falsifiability gates)
+tests/                                # 19 files, 488 tests (planner/bench split per-concern; test_accuracy falsifiability gates)
 docs/                                 # 18 guides (~12,400 lines total)
 resources/prompts/                    # Legacy banter_prompts.txt (not used in benchmarking)
 ```
@@ -261,25 +261,26 @@ The planner is no longer limited to the 7 bundled registry models. `plan --model
 ## Testing
 
 ```bash
-pytest tests/ -v                    # 483 total tests
+pytest tests/ -v                    # 488 total tests
 pytest tests/ --cov=src             # With coverage
 ```
 
-**Layout** (483 tests, 19 files -- planner/bench split per-concern after 0.3.0):
+**Layout** (488 tests, 19 files -- planner/bench split per-concern after 0.3.0):
 
-- **Planner** (161): test_planner_models.py (58 - 7 predictive models: VRAM/throughput/
-  quality/latency/scaling/cost/safety, incl. roofline + KV-batch concurrency),
-  test_planner_engine.py (55 - gate search, N-replica x B-batch, Pareto, variance guard,
-  provenance), test_planner_cli.py (18), test_planner_core.py (17 - serialization,
-  find_models_for_size), test_accuracy.py (13 - numerical falsifiability gates)
+- **Planner** (163): test_planner_models.py (60 - 7 predictive models: VRAM/throughput/
+  quality/latency/scaling/cost/safety, incl. roofline + KV-batch concurrency +
+  shared FP16-baseline resolver), test_planner_engine.py (55 - gate search,
+  N-replica x B-batch, Pareto, variance guard, provenance), test_planner_cli.py (18),
+  test_planner_core.py (17 - serialization, find_models_for_size), test_accuracy.py
+  (13 - numerical falsifiability gates)
 - **Model-agnostic** (43): test_resolver.py (25 - ModelSpec, registry/Ollama/HF/manual +
   cache), test_discovery.py (12 - suggest/catalog), test_measure.py (6 - measure-on-demand)
 - **Safety** (54): test_safety.py - refusal lookup, RTSI tiers, identity resolution
 - **Bench** (70): test_bench_metrics.py (28), test_bench_backends.py (20 - Ollama/vLLM/TGI),
   test_bench_runner.py (17 - runner, sweeps, resilience), test_bench_cli.py (5)
 - **Refit/Eval/Report/Compare** (141): test_refit.py (47 - Bayesian blend + per-key
   weighting + validation), test_eval.py (42), test_report.py (32), test_compare.py (20)
-- **CLI hardening** (9): test_cli_fail_loud.py - clean errors + exit codes, no raw tracebacks
+- **CLI hardening** (12): test_cli_fail_loud.py - clean errors + exit codes, no raw tracebacks
 - **Monitoring** (5): test_monitoring.py - SLO eval, log parsing, thread-safe aggregation,
   recommender, monitor lifecycle
 

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "chimeraforge"
-version = "0.6.1"
+version = "0.6.2"
 description = "LLM deployment optimizer (performance, cost, and safety) — backed by ~204,000 real measurements on consumer GPUs"
 readme = "README.md"
 requires-python = ">=3.10"

src/chimeraforge/__init__.py

@@ -5,4 +5,4 @@
 inference deployments.
 """
 
-__version__ = "0.6.1"
+__version__ = "0.6.2"

src/chimeraforge/planner/models.py

@@ -343,46 +343,67 @@ class QualityModel:
         "concerning": -15.0,
     }
 
-    def predict(self, model: str, quant: str) -> float:
+    def _fp16_baseline(self, model: str, family: str | None = None) -> float | None:
+        """Resolve an FP16-equivalent quality baseline, or None if unanchored.
+
+        One chain, shared by predict()/estimate()/quality_tier() so the reported
+        quality, its provenance, and the tier can never diverge:
+          1. measured FP16 baseline (``fp16_baselines``)
+          2. an explicit ``model|FP16`` lookup entry
+          3. the model's own highest-precision measured quant -- TR125 uses Q8_0
+             as the llama3.1-8b baseline (8B FP16 weights don't fit the test GPU,
+             so no FP16 row exists); more specific than a family mean, so it wins
+          4. the family's mean FP16 baseline (off-registry models)
+        """
+        fp16 = self.fp16_baselines.get(model)
+        if fp16 is None and f"{model}|FP16" in self.lookup:
+            fp16 = self.lookup[f"{model}|FP16"]
+        if fp16 is None:
+            prefix = f"{model}|"
+            best_bpw = -1.0
+            for lk, val in self.lookup.items():
+                if lk.startswith(prefix):
+                    bpw = QUANT_BPW.get(lk[len(prefix) :], 0.0)
+                    if bpw > best_bpw:
+                        best_bpw, fp16 = bpw, val
+        if fp16 is None and family is not None:
+            same_family = [
+                v for m, v in self.fp16_baselines.items() if MODEL_FAMILY.get(m) == family
+            ]
+            if same_family:
+                fp16 = sum(same_family) / len(same_family)
+        return fp16
+
+    def predict(self, model: str, quant: str, family: str | None = None) -> float:
         """Predict composite quality [0, 1]."""
         key = f"{model}|{quant}"
         if key in self.lookup:
             return self.lookup[key]
-        fp16 = self.fp16_baselines.get(model)
+        fp16 = self._fp16_baseline(model, family)
         if fp16 is None:
-            # Infer FP16 baseline from lookup if available
-            fp16_key = f"{model}|FP16"
-            if fp16_key in self.lookup:
-                fp16 = self.lookup[fp16_key]
-        if fp16 is not None:
-            if quant == "FP16":
-                return fp16
-            delta = self.quant_deltas.get(quant, 0.0)
-            return max(0.0, min(1.0, fp16 + delta))
-        return 0.5
+            return 0.5
+        if quant == "FP16":
+            return fp16
+        delta = self.quant_deltas.get(quant, 0.0)
+        return max(0.0, min(1.0, fp16 + delta))
 
     def estimate(self, model: str, quant: str, family: str | None = None) -> tuple[float, str]:
         """Predict quality and report provenance: measured | estimated | unknown.
 
         - ``measured``: a direct (model, quant) lookup hit (TR-backed).
-        - ``estimated``: derived from the model's own FP16 baseline + quant delta,
-          or, for an off-registry model, the mean FP16 baseline of its family.
+        - ``estimated``: derived from an FP16-equivalent baseline (:meth:`_fp16_baseline`)
+          + quant delta -- the model's own FP16 row, its highest measured quant, or
+          its family mean, in that priority.
         - ``unknown``: no basis -- returns the neutral 0.5 prior, flagged so the
           caller never mistakes a guess for data.
+
+        Shares :meth:`_fp16_baseline` with :meth:`quality_tier`, so the reported
+        quality and its tier are always computed off the same baseline.
         """
         if f"{model}|{quant}" in self.lookup:
             return self.lookup[f"{model}|{quant}"], "measured"
 
-        fp16 = self.fp16_baselines.get(model)
-        if fp16 is None and f"{model}|FP16" in self.lookup:
-            fp16 = self.lookup[f"{model}|FP16"]
-        if fp16 is None and family is not None:
-            same_family = [
-                v for m, v in self.fp16_baselines.items() if MODEL_FAMILY.get(m) == family
-            ]
-            if same_family:
-                fp16 = sum(same_family) / len(same_family)
-
+        fp16 = self._fp16_baseline(model, family)
         if fp16 is None:
             return 0.5, "unknown"
         if quant == "FP16":
@@ -393,32 +414,12 @@ def estimate(self, model: str, quant: str, family: str | None = None) -> tuple[f
     def quality_tier(self, model: str, quant: str, family: str | None = None) -> str:
         """Classify quality drop into a tier.
 
-        Family-aware, mirroring :meth:`estimate`: an off-registry model whose
-        family matches the registry derives its FP16 baseline (and predicted
-        quality) from the family mean, so the tier is consistent with the
-        reported quality instead of silently collapsing to ``unknown``.
+        Uses the same :meth:`_fp16_baseline` chain as :meth:`estimate` (measured
+        FP16 -> highest measured quant -> family mean), so the tier is always
+        consistent with the reported quality instead of one collapsing to
+        ``unknown`` while the other resolves a baseline.
         """
-        fp16 = self.fp16_baselines.get(model)
-        if fp16 is None and f"{model}|FP16" in self.lookup:
-            fp16 = self.lookup[f"{model}|FP16"]
-        if fp16 is None:
-            # No FP16 measurement (e.g. an 8B model whose FP16 weights don't fit
-            # the test GPU) -- anchor to the model's own highest-precision measured
-            # quant, matching TR125 (Q8_0 baseline for llama3.1-8b). More specific
-            # than a family mean, so it takes precedence.
-            prefix = f"{model}|"
-            best_bpw = -1.0
-            for lk, val in self.lookup.items():
-                if lk.startswith(prefix):
-                    bpw = QUANT_BPW.get(lk[len(prefix) :], 0.0)
-                    if bpw > best_bpw:
-                        best_bpw, fp16 = bpw, val
-        if fp16 is None and family is not None:
-            same_family = [
-                v for m, v in self.fp16_baselines.items() if MODEL_FAMILY.get(m) == family
-            ]
-            if same_family:
-                fp16 = sum(same_family) / len(same_family)
+        fp16 = self._fp16_baseline(model, family)
         if fp16 is None or fp16 <= 0:
             return "unknown"
         # Predicted quality consistent with estimate(): direct lookup, else the

tests/test_planner_models.py

@@ -185,6 +185,20 @@ def test_quality_tier_no_fp16_anchors_to_best_quant(self, bundled_models):
         assert q.quality_tier("llama3.1-8b", "Q4_K_M") != "unknown"
         assert q.quality_tier("llama3.1-8b", "Q2_K") != "unknown"
 
+    def test_estimate_and_tier_share_baseline(self, bundled_models):
+        # Regression (#4 follow-up): estimate() and quality_tier() must resolve
+        # the SAME FP16 baseline. llama3.1-8b has no measured FP16 row, so both
+        # anchor to the highest measured quant (Q8_0). estimate() must then report
+        # 'estimated' -- not the 0.5/'unknown' prior, which wrongly ranked FP16
+        # (highest precision) below Q2_K -- and the tier must be a real class
+        # consistent with that quality.
+        q = bundled_models.quality
+        val, src = q.estimate("llama3.1-8b", "FP16")
+        assert src == "estimated"
+        assert val == pytest.approx(q.predict("llama3.1-8b", "FP16"))
+        assert val >= q.estimate("llama3.1-8b", "Q2_K")[0]
+        assert q.quality_tier("llama3.1-8b", "FP16") != "unknown"
+
     def test_unknown_model_returns_default(self):
         m = QualityModel()
         q = m.predict("nonexistent", "FP16")