170
quantization
Fewer bits per weight. Smaller and faster. Measure the quality drop. Do not assume it is free.
What is quantization?
Quantization stores numbers in fewer bits (for example 8 or 4 instead of 16). The model gets smaller and faster and slightly less accurate. The drop is task-specific. You measure it on the holdout, not on a vibe.
Why does quantization matter when vibe coding?
Models quantize to fit a laptop and ship without rerunning the eval. Name the before-and-after metric.
How do you do quantization?
Run the same holdout before and after. If the metric drop is unacceptable, quantize less or not at all. Latency is the other number. Report both.
How do you ask a model for quantization?
Quantize (model) only if the holdout metric stays within (tolerance) and latency improves. Report both numbers against the unquantized model. Do not quantize blind.
What goes wrong with quantization?
Quantizing after a calibration set that does not look like production. The speed is real and the errors are not the ones you measured.