Find out exactly which local LLMs your PC can run — VRAM math, best quantization, tok/s estimates, and bottlenecks. No model downloads required.
- Cross-platform — Windows, macOS, and Linux
- Multi-source GPU detection — NVML, nvidia-smi, ROCm, sysfs, PowerShell, lspci
- VRAM estimator — weights + KV cache + overhead (llama.cpp-aligned)
- Model catalog — 12 popular open models from 3B to 72B
- Rich TUI — hardware cards, VRAM fill bars, bottleneck tips
- JSON export — for scripting and CI
git clone https://github.com/doteyeso-ops/ai-benchmark.git
cd ai-benchmark
python -m venv .venv
# Windows
.venv\Scripts\activate
# macOS / Linux
source .venv/bin/activate
pip install -e .Optional NVIDIA telemetry (more accurate VRAM on Windows):
pip install -e ".[nvidia]"ai-benchmark # Rich TUI (default)
ai-benchmark --plain # Simple tables (CI / logs)
ai-benchmark --json # Machine-readable output
ai-benchmark -c 32768 # Evaluate at 32K context
ai-benchmark --micro-bench # Include quick CPU probe| Section | Details |
|---|---|
| Hardware | CPU, RAM, GPU/VRAM, Apple Silicon unified memory |
| Compatibility | Per-model fit: OK / tight / CPU offload / no fit |
| Throughput | Estimated tok/s from published hardware profiles |
| Bottlenecks | Primary constraint + actionable recommendations |
VRAM = model weights + KV cache + framework overhead
- Weights:
params x bytes_per_param(Q4_K_M ~ 0.625 B/param) - KV cache:
2 x layers x kv_heads x head_dim x context x bpp(GQA-aware) - Overhead: ~1.2 GB + 512 MB system reserve
Throughput uses lookup tables (RTX 3060/3090/4090/5090, Apple M-series) scaled by model size. For ground truth, use llama-bench.
pip install -e ".[dev]"
pytest tests/ -q
python scripts/generate_screenshot.py # Regenerate README image- GGUF metadata parser for custom models
- Optional
llama-benchintegration - HTML report export
- llama-bench
- Will It Fit? — VRAM estimator
- edgemark — multi-model benchmark
MIT — see LICENSE.