diff --git a/benchmark_results/jetson_agx_xavier/FINDINGS.md b/benchmark_results/jetson_agx_xavier/FINDINGS.md new file mode 100644 index 0000000..2c4181d --- /dev/null +++ b/benchmark_results/jetson_agx_xavier/FINDINGS.md @@ -0,0 +1,82 @@ +# Jetson AGX Xavier MTP Benchmark: Q3_K_M vs APEX I-Compact + +## Device + +| Spec | Value | +|---|---| +| Device | NVIDIA Jetson AGX Xavier (MiiVii, Tegra194) | +| GPU | Volta SM72, 512 CUDA cores, 7.6 TFLOPS FP16 | +| Memory | 32GB unified (CPU+GPU), ~135 GB/s bandwidth | +| Power | 30W | +| OS | Ubuntu 20.04 aarch64 | +| CUDA | 11.4 | +| Software | llama.cpp master (SHA 571d0d54), built with CUDA | + +## Models Tested + +Both quantizations of `Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved` (the uncensored fine-tune that preserves all 20 MTP layers): + +- **Q3_K_M (K-quant, bartowski)**: 17.3 GB — standard K-quant baseline +- **APEX I-Compact (SC117/cvgro)**: 17.0 GB — APEX layer-wise precision gradient (5 + 5 edge layers at higher precision, middle 30 layers compressed) +- ~~APEX I-Mini~~: Not available at time of writing (404 on both SC117 and cvgro repos). Original PR data retained for reference only. + +## Methodology + +Each configuration tested with **5 runs** (128 tokens each, fixed prompt, seed=42). Results reported as **mean ± std dev** over 5 runs. Sequential test order: Q3_K_M baseline → Q3_K_M MTP → APEX baseline → APEX MTP (same session, no reboot between config changes). + +> **Note on baseline drift**: The original PR single-run data recorded Q3_K_M baseline at 15.96 t/s. Our 5-run mean is 19.32 t/s — a ~21% shift attributed to environmental differences (thermal state, system load). This discrepancy does not affect within-session comparisons (baseline vs MTP for the same model), but cross-session speedup ratios are unreliable. + +## Baseline Results (no speculative decoding) + +| Quant | Size | Mean ± σ | CV | vs Q3_K_M | +|---|---|---|---|---| +| Q3_K_M | 17.3 GB | **19.32 ± 0.02 t/s** | 0.1% | baseline | +| **APEX I-Compact** | 17.0 GB | **19.97 ± 0.05 t/s** | 0.2% | **+3.4%** | + +Both quantizations show extremely low variance (≤0.2% CV), confirming test reproducibility. APEX I-Compact's **+3.4% baseline advantage** over Q3_K_M is small but consistent (all 5 runs above Q3_K_M's max). This may reflect APEX's selective precision allocation improving compute efficiency on memory-bandwidth-bound AGX Xavier. + +> Original PR reported Q3_K_M at 15.96 t/s and APEX I-Compact at 15.97 t/s (essentially tied). Our multi-run data shows a small but reproducible APEX advantage. + +> APEX I-Mini original PR baseline: 14.67 t/s (single run, not verified). + +## MTP Speculative Decoding Results (`--spec-type draft-mtp --spec-draft-n-max 2`) + +| Quant | Mean ± σ | CV | Speedup vs baseline | MTP Accept Rate* | +|---|---|---|---|---| +| Q3_K_M | **22.47 ± 0.40 t/s** | 1.8% | **+16.3%** | 97.7% | +| **APEX I-Compact** | **24.85 ± 0.07 t/s** | 0.3% | **+24.4%** | 69.8% | + +> *Accept rates from original PR (single run). Q3_K_M speedup of 16.3% vs our baseline (original PR: +45% vs 15.96 t/s). APEX speedup of 24.4% (original PR: +19.0% vs 15.97 t/s). + +**Key observation**: APEX I-Compact achieves **higher absolute MTP throughput (24.85 t/s) and higher speedup (+24.4%)** than Q3_K_M MTP (22.47 t/s, +16.3%), despite a significantly lower acceptance rate (69.8% vs 97.7%). The directly observed result stands on its own — the acceptance rates are carried over from the original PR (different session, ~21% Q3 baseline drift), so no causal formula can be reliably derived from cross-session measurements. + +## Key Finding + +On this Jetson AGX Xavier test system, **APEX I-Compact outperforms Q3_K_M in both baseline (+3.4%) and MTP (+10.6% absolute)** throughput (directly observed, same test session). + +### Practical implications + +| Metric | Q3_K_M | APEX I-Compact | Winner | +|---|---|---|---| +| Baseline t/s | 19.32 | 19.97 | **APEX** (+3.4%) | +| MTP t/s | 22.47 | 24.85 | **APEX** (+10.6%) | +| Speedup ratio | +16.3% | +24.4% | **APEX** | +| Model size | 17.3 GB | 17.0 GB | **APEX** (−0.3 GB) | +| Accept rate | 97.7% | 69.8% | Q3_K_M | +| TPS stability (CV) | 1.8% | 0.3% | **APEX** | + +## Caveats + +1. **Single device (N=1)**: Results are device-specific and may not generalize. +2. **Single prompt, fixed length**: Only tested with one prompt (180 chars) and 128 output tokens. +3. **Thermal ordering**: APEX was tested after Q3_K_M — the device may have been in a different thermal state. +4. **Accept rate not independently measured**: Values from original PR (different session, ~21% Q3 baseline drift). Accept rates on our hardware may differ significantly — direct measurement would be needed for a proper causal model. +5. **No APEX I-Mini**: The 13.3 GB variant is no longer available, limiting the comparison to similar-size models only. +6. **Causal mechanism speculative**: The link between APEX's layer-wise precision gradient and accept rate reduction is inferred, not experimentally validated. + +## Suggested Next Steps + +1. **Direct acceptance rate measurement** via llama.cpp's `--print-acceptance` or per-token logprobs comparison +2. **Test APEX I-Balanced/I-Quality** on devices with ≥48 GB unified memory to see if gentler compression preserves accept rates +3. **Cross-validate with different prompts and sequence lengths** +4. **Run reverse order** (APEX first, Q3_K_M second) to isolate thermal effects diff --git a/benchmark_results/jetson_agx_xavier/mtp_results.txt b/benchmark_results/jetson_agx_xavier/mtp_results.txt new file mode 100644 index 0000000..7336176 --- /dev/null +++ b/benchmark_results/jetson_agx_xavier/mtp_results.txt @@ -0,0 +1,28 @@ +# MTP Speculative Decoding Benchmarks on NVIDIA Jetson AGX Xavier +# Device: Jetson AGX Xavier (Volta, 512-core, 32GB unified memory, ~135 GB/s, 30W) +# Software: llama.cpp master (SHA 571d0d54), CUDA 11.4, Ubuntu 20.04 aarch64 +# MTP config: --spec-type draft-mtp --spec-draft-n-max 2 +# Benchmark: 128 tokens generated after fixed prompt (180 chars), 5 runs each, seed=42 +# Base model: Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved +# +# Format: +# model size_gb mean_tps std_tps min_tps max_tps n_runs +# For single-run values (original PR): mean_tps is the single value, n_runs=1 +# tps = tokens per second (timings.predicted_per_second from API) +# +# --- All 5-run data collected 2026-07-24, same prompt+seed, sequential runs --- +# +# Q3_K_M (bartowski), baseline (no speculative decoding) +Q3_K_M_baseline 17.3 19.32 0.02 19.29 19.34 5 +# Q3_K_M (bartowski), MTP (--spec-type draft-mtp --spec-draft-n-max 2) +Q3_K_M_mtp_n2 17.3 22.47 0.40 21.76 22.73 5 +# +# APEX I-Compact (SC117/cvgro), baseline (no speculative decoding) +APEX_I-Compact_baseline 17.0 19.97 0.05 19.92 20.02 5 +# APEX I-Compact (SC117/cvgro), MTP (--spec-type draft-mtp --spec-draft-n-max 2) +APEX_I-Compact_mtp_n2 17.0 24.85 0.07 24.76 24.91 5 +# +# --- Original PR single-run data (kept for reference) --- +# Note: Q3_K_M baseline 15.96 t/s (single run) vs our 19.32 t/s (5-run mean) +# Baseline drift ~21% attributed to thermal/system load differences +# MTP accept rates from original PR (different session): Q3_K_M: 97.7%, APEX I-Compact: 69.8%, APEX I-Mini: 70.5%