quantization
Quantization represents model values with reduced numerical precision.
Use this when A local model exceeds the available memory or latency budget.
without the term
“The model I want to run locally doesn't fit in my laptop's memory. Is there a way to make it smaller without it getting a lot dumber?”
with the term · How do you ask a model for quantization?
Evaluate quantization for this model on the target device. Compare memory, latency, and task quality against the original representation.
What is quantization?
Weights are stored with fewer bits, which can cut memory and sometimes latency. The quality and speed effects depend on the model and runtime, so measure them on the target device.
Why does quantization matter when vibe coding?
A smaller model file does not guarantee faster inference or acceptable task quality.
How do you do quantization?
Compare supported representations on the target runtime with representative tasks.
What goes wrong with quantization?
Testing quality on a tiny calibration sample that misses important production inputs.