Stop guessing your inference settings. Measure them.
QuantMap is a measurement and reporting system for local LLM inference benchmarking. It runs structured campaigns that sweep server parameters (thread counts, batch sizes, GPU layer offloading) and collects structured telemetry for evidence-bound analysis and reporting.
The Phase 1 Trust Bundle and Phase 1.1 stabilization pass are stable after real-workflow validation. QuantMap now treats snapshot-first historical identity, methodology evidence, layered runtime/report state, and non-misleading export/report trust behavior as the foundation for future work.
The active focus is Phase 3: Platform Generalization. Phase 2 Operational Robustness and Phase 2.1 Settings/Environment Bridge are closed after validation. Phase 3 begins with boundary-aware telemetry/provider design, not scattered provider conditionals in existing high-blast-radius modules.
Normal local work assumes the repo .venv already exists. On open, VS Code has a repo task named QuantMap: Dev Contract Preflight that runs the required startup check:
.\.venv\Scripts\python.exe .agent\scripts\helpers\verify_dev_contract.py --quickIf that check passes, keep using .\.venv\Scripts\python.exe for repo-health commands. Shell activation and bare python/pip are not the local contract.
If the quick check fails because .venv is missing or drifted, folder-open preflight performs one self-heal rebuild and reruns the quick check.
First open still requires VS Code workspace trust before any workspace task can run. After trust is granted, checked-in workspace settings allow the preflight task to run automatically on folder open.
For a fresh clone or explicit repair, rebuild the repo .venv from the approved DevStore Python target:
& D:\.store\mise\data\installs\python\3.13.13\python.exe -m venv .venv
.\.venv\Scripts\python.exe -m pip install --no-user --upgrade pip
.\.venv\Scripts\python.exe -m pip install --no-user -e '.[dev]'
.\.venv\Scripts\python.exe .agent\scripts\helpers\verify_dev_contract.py --full
.\.venv\Scripts\python.exe -m pytest -qIf .venv is locked or drifted in VS Code, run the QuantMap: Repair Dev Venv task. It stops workspace-scoped Python/Ruff/mypy holders, rebuilds .venv from DevStore Python 3.13.13, reinstalls dev dependencies, and reruns the full contract check. Development scaffolding such as .agent, CI checks, and editor defaults must not become QuantMap runtime behavior.
QuantMap is built on the principle that benchmarking is a forensic science.
- A Monitored Environment: A system that observes and logs background interference to ensure data transparency.
- An Evidence-Bound Narrator: A briefing engine that only speaks when statistical margins are significant.
- A Durable Forensic Record: A persistent, traceable history of every request, response, and thermal event.
- A Magic Optimizer: It will not "fix" a bad configuration; it will provide evidence that it is sub-optimal.
- An Inference Engine: It calls
llama-server. It does not perform the calculations itself. - A vibes-based ranker: If Config A is 1% faster but 5x more unstable, it will not win.
- It cannot make invalid comparisons valid: If you compare two campaigns using different methodologies, the result is technically a mismatch.
- It cannot repair bad raw data: No amount of
rescorewill fix a run that was corrupted by thermal throttling or background indexing. - It cannot infer missing telemetry: If HWiNFO is not running, thermal events are recorded as "Unknown."
We separate the tool from the rules.
| Feature | Software Updates | Methodology Updates |
|---|---|---|
| Focus | CLI ergonomics, diagnostics, reports. | Scoring weights, gates, thresholds. |
| Impact | Changes how you see the data. | Changes what the data concludes. |
| Historical Data | Raw measurements are never altered. | Rescoring creates a new interpretation floor. |
Note
What Changed vs. What Did Not:
- Software changes may affect the UI, packaging, or diagnostic speed. Historical outcomes remain untouched.
- Methodology changes (Registry/Profile) affect winner selection and comparison validity.
- Rescoring a campaign under persisted historical methodology is snapshot-locked. Current-input rescoring is an explicit migration-like mode and must be labeled as such.
A standard successful run follows this clinical sequence:
# 1. Setup and Pulse Check
quantmap init
quantmap doctor
quantmap self-test
# 2. Execution (Dry-run first to verify budget)
quantmap run --campaign C01 --mode quick --dry-run
quantmap run --campaign C01 --mode quick
# 3. Analysis and Briefing
quantmap explain C01If results are surprising or the tool behavior is unexpected, run these five commands in sequence to gather a forensic baseline:
quantmap about— Who am I and what are my rules?quantmap status— Is my lab currently healthy?quantmap doctor— Is my background currently silent?quantmap self-test— Is my core math still valid?quantmap export <id> --strip-env— Generate a redacted case file for peer review.
- Operator Playbooks — How to actually think and operate with this tool.
- Quickstart — Fast command-oriented onboarding for common workflows.
- Trust Surface — How QuantMap proves its findings.
- System Architecture — The technical design of the pipeline.
Built by Mad-Labs42 — because guessing is not engineering.