vllm-tune

A benchmark harness and lab notebook for tuning local vLLM inference.

The project measures quality, latency, and long-prefill behavior against an already-running OpenAI-compatible endpoint. It changes per-request sampling parameters only; it does not restart or reconfigure the server.

This is deliberately a lab tool, not a universal optimizer. The included reports came from a local dual-RTX 3090 setup and should be treated as evidence about that setup, not defaults for every model or GPU.

What it does

• Runs structural gates before quality scoring.
• Uses an external judge; the model under test never judges itself.
• Keeps max_tokens fixed so optimization cannot win by producing shorter answers.
• Supports paired, seeded trials for deterministic models.
• Supports interleaved unseeded A/B runs for non-deterministic or speculative-dechome-automationg models.
• Characterizes run-to-run noise before accepting a winner.
• Confirms search winners with repeated runs and a mean + 2σ threshold.
• Measures long-prefill TTFT and queue behavior under mixed request sizes.
• Simulates admission-policy shapes from measured service times.

Install

Python 3.11 or newer is required.

python -m venv .venv
. .venv/bin/activate
pip install -e .

For development:

uv sync --group dev
make ci

Configure

Copy one of the files under config/ and point it at your endpoints. The examples use localhost and intentionally contain no credentials.

endpoint: http://localhost:8000/v1/chat/completions
model: model-under-test

judge:
  endpoint: http://localhost:8001/v1/chat/completions
  model: independent-judge

The target and judge should be different models. A Claude CLI judge is also supported when no suitable independent local judge exists.

Sampling workflows

Seed-deterministic models

.venv/bin/python scripts/benchmark.py --config config/sampling_params.yaml
.venv/bin/python scripts/noise_run.py --config config/sampling_params.yaml --runs 20
.venv/bin/python scripts/optimize.py --config config/sampling_params.yaml --freeze-baseline
.venv/bin/python scripts/optimize.py --config config/sampling_params.yaml --study --trials 100
.venv/bin/python scripts/optimize.py --config config/sampling_params.yaml --confirm --top 3 --reps 10

Non-deterministic models

Speculative dechome-automationg and some model architectures can return different output for the same seed. In that case, use interleaved unseeded A/B runs:

.venv/bin/python scripts/ab_test.py --config config/deepthink_pp.yaml --phase gen --reps 5
.venv/bin/python scripts/ab_test.py --config config/deepthink_pp.yaml --phase judge
.venv/bin/python scripts/ab_test.py --config config/deepthink_pp.yaml --phase report

Agentic prefill experiments

Generate the synthetic long-context corpus locally:

.venv/bin/python benchmarks/build_longctx.py

Run a direct endpoint bakeoff:

.venv/bin/python scripts/run_agentic_prefill_bakeoff.py \
  --endpoint http://localhost:8000/v1/chat/completions \
  --model model-under-test

scripts/simulate_agentic_prefill_admission.py compares admission-policy shapes using the normalized measurements in results/fixtures/agentic_prefill_calibration.jsonl. Its output is diagnostic: simulation findings must be validated against live traffic before changing production.

.venv/bin/python scripts/simulate_agentic_prefill_admission.py

Generated simulation records and reports are written under results/generated/.

Evidence

The full experiment index is in results/README.md. It covers throughput scaling, admission simulation, live prefill-fairness corrections, NVLink controls, parser comparison, and sampling studies. Curated numeric datasets under results/data/ support the published tables. Raw responses, judge caches, SQLite studies, generated corpora, logs, and machine-specific operational files remain intentionally excluded.

Principles

1. Quality is the objective; speed and structure are gates.
2. Do not tune max_tokens.
3. Do not let the target judge itself.
4. Prove seed determinism before using paired seeds.
5. Characterize noise before trusting small deltas.
6. Treat local benchmark results as local evidence, not universal truth.
7. Restore and verify the serving stack after any separate server-side experiment.

License

MIT