fix: resolve issues #4 (quality_tier unknown) and #5 (quick-start httpx)#8
Open
Sahil170595 wants to merge 1 commit into
Open
fix: resolve issues #4 (quality_tier unknown) and #5 (quick-start httpx)#8Sahil170595 wants to merge 1 commit into
Sahil170595 wants to merge 1 commit into
Conversation
#4: quality_tier returned "unknown" for llama3.1-8b on every config -- its FP16 was never measured (16GB exceeds the RTX 4080), so it has no FP16 baseline and is the only member of its family (family fallback finds nothing). TR125 handled this by using Q8_0 as the baseline; quality_tier now mirrors that, anchoring to the model's highest-precision measured quant when no FP16 baseline exists. No fabricated data -- the Q8_0 baseline is the documented TR125 methodology. Verified: `plan --model-size 8b --json` now reports negligible/acceptable tiers instead of unknown. #5: docs/quick_start.md installed deps via `pip install -r requirements.txt`, but requirements.txt is a comment-only stub (deps live in pyproject), so it installed nothing and the agent demo failed with `ModuleNotFoundError: httpx`. Switched to `pip install -e ".[bench]"` (provides httpx) and pointed the clone URL at the real repo (was `your-org` placeholder). 487 tests pass (+1 regression); src/ lint clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4. Closes #5.
Stacked on #7 (the 0.6.1 branch) — base is
fix/0.6.1-deps-httpx, so this diff shows only the issue fixes. It auto-retargets tomainonce #7 merges.#4 —
quality_tieralwaysunknownforllama3.1-8bConfirmed the report:
llama3.1-8bhas 6 quantized quality entries but no FP16 baseline and no|FP16lookup key, and it's the solellama3.1-family member, so the family fallback finds nothing →unknownon every config.Why the reporter's suggested fix (add the measured FP16 baseline) is impossible: TR125 source states FP16 for this model was never measured — 16 GB exceeds the RTX 4080 — and the research used Q8_0 as the baseline instead (
experiments/research/tr125/phase2/analyze.py:15,generate_report.py:1005). Fabricating an FP16 value would break provenance.Fix (honest, mirrors TR125): when no FP16 baseline exists,
quality_tieranchors to the model's highest-precision measured quant (Q8_0). No data invented.Verified —
chimeraforge plan --model-size 8b --hardware "RTX 4090 24GB" --request-rate 2.0 --json(the issue's exact command):quality_tierunknown(all 15 configs)negligible/acceptable#5 — quick-start agent demo fails with
ModuleNotFoundError: httpxRoot cause is subtler than "httpx missing from deps":
docs/quick_start.mdtold userspip install -r requirements.txt, butrequirements.txtis a comment-only stub (deps live inpyproject.toml), so it installs nothing — and the demo hard-importshttpx. Fixed the doc topip install -e ".[bench]"(provideshttpx) and corrected theyour-orgplaceholder clone URL.Verification
llama3.1-8btier is neverunknown);src/lint clean.