Quantization Lab
Fewer bits. Visible error.
Move eight values onto an integer grid. See every rounded code, clipped value, and reconstructed number.
Live result
The rounding ledger
Scroll the diagram sideways. Exact values appear below.
This is per-tensor linear quantization of a tiny vector. It does not model GGUF blocks, NF4, optimizer memory, model quality, or packed-file overhead beyond the stated metadata.
The explanation
A small grid has to make choices.
The input is first converted to FP16 so the baseline is an actual half-precision vector. The displayed input and FP16 columns expose that conversion before integer quantization begins. [Q3]
The scale sets the spacing between reconstructed values. A wider calibration range admits larger magnitudes but leaves more space between grid points. Shrinking the range increases saturation risk. The clipping preset shows this trade-off directly. [Q1]
Zero point chooses the integer code for real zero. The affine scheme includes zero in its calibration interval; the symmetric scheme fixes zero point at zero. Reconstruction brings each code back to the same real-valued scale. [Q2]
Four bits is one quarter of the FP16 payload, but a tiny tensor still needs metadata. This demo adds five metadata bytes and reports both payload and total. Lower numerical error does not establish better downstream model quality.
qmax=2^(b-1)-1; qmin=-qmax; s=max(|calMin|,|calMax|)/qmax; z=0
Open Asset Factory
Take the explanation with you.
Three independent diagrams. Editable source and high-resolution PNGs, with assumptions printed on the image.
Rounding versus clipping
Two distinct ways a reconstructed value can differ.
Sources checked 2026-09-07
Claims you can inspect.
Formula derivations and teaching assumptions are documented separately.
Download claim ledgerOpenEXR documents half precision with one sign bit, five exponent bits, ten fraction bits, and subnormal values.
Primary source ↗Only finite values up to 65504 are accepted here; conversion uses nearest-even rounding as an explicit policy.
ONNX QuantizeLinear specifies scale, zero point, saturation, and nearest-even rounding. Signed INT4 spans -8 through 7.
Primary source ↗The symmetric teaching scheme deliberately leaves the most negative code unused.
ONNX DequantizeLinear defines reconstruction as (integer − zero point) × scale.
Primary source ↗Floating-point implementations may introduce additional rounding.