Llama.cpp for macOS Notes

• -ngl 99: Offloads all layers to the GPU to utilize Unified Memory.

• -fa on: Enables Flash Attention to save memory bandwidth during long conversations.

• -b 1024: Sets batch size (optimized for M4 Air thermals).

• -ub 512: Sets micro-batch size.

• -c (Context Size): Practical sizes like 8192 or 16384 are recommended for interactive chat to save memory.

• --parallel: Sets the number of concurrent request slots.

• K vs V Quantization: The Key (K) cache is more sensitive to quantization than the Value (V) cache. Use K8V4 (--cache-type-k q80 --cache-type-v q40) for optimal memory savings without severe quality degradation.

• Memory Mapping (--mmap vs --mlock): --mmap is enabled by default and recommended for Apple Silicon. Only use --mlock if the model fits comfortably within 70% of your total RAM to avoid system freezing.

• Mixture of Experts (--cmoe): Avoid using -cmoe or -ncmoe flags on Apple Silicon. Because of Unified Memory, forcing expert layers onto the CPU only slows down processing without saving memory.

• Metal is now enabled by default when building on macOS.

• Memory bandwidth (120 GB/s on M4 Air) is the main bottleneck for small models, so keep batch sizes moderate to prevent thermal throttling.

  • PP is compute bound

• Use MLX for raw speed on small models (<12B) if you don't need massive context windows.

• --priority: Sets the scheduling priority of the process (0-3). Setting to 2 reduces the chance of inference threads being interrupted on macOS.

• Quantization Baseline: Q4_K_XL is generally considered the sweet spot for speed and quality on Apple Silicon.

  • prefer Q5_K_M for near lossless

---

See Details.md for further elaboration

See lm-studio for an example

See resources for external sources

See articles

---

Notes above were written by Siri AI via Apple Intelligence