This is a basic application that allows you to visualize the precision of a floating point number with various mantissa and exponent sizes along the real number line.
It is based off Olli Etuaho's js app (code) that has functions to simulate floating point operations at arbitrary precision, but converted to Python and using matplotlib for visualization along with common precisions implemented in Nvidia GPU or other hardware.
The purpose is to visual precision and dynamic range, allowing zoom across the real number line.
All graphs are of the real number line (x) against the quantized dtype (y). This is only a quick approximation for visualization purposes, and actual behavior for casting inputs, calculations, and rounding of accumulations will depend on the specific hardware op or software stack being used.
- Python 3 (written in 3.10)
- tkinter
- matplotlib
- numpy
Launch app with
python precision.py
Try out the zoom and recenter buttons to compare how different numbers are represented. For instance, try zooming in on the number 1.0 (1e0) and see how the precisions compare as you zoom in, or zoom to ~1e5 to see where FP16 range is limited (65,504). Toggle different dtypes to compare them with less clutter.
The simulation for FP8 is only trivially implemented and probably needs more investigation of the FP8 paper and the actual hardware implementation and behavior.
Low bit formats, particularly 9-bit and 4-bit variants, are often used within microscaling formats for weight quantization whether post-training quanization or pre-training model design. For information on how low-bit data types are used in practice by micro-scaling, check out these resources on microscaling formats:
Microscaling Data Formats for Deep Learning
Reverse Engineering GGUF (video) (discussing ggml's llama.cpp)
Pretraining Large Language Models with NVFP4
What's MXFP4? (used by gpt oss)
SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models and github
Recipes for Pre-training LLMs with MXFP8
8-bit Optimizers via Block-wise Quantization (basis of bitsandbytes which includes optimizer state quantization such as AdamW8bit)
