INFERENCE OPTIMIZATION: WHEN TO QUANTIZE, DISTILL, OR JUST USE A SMALLER MODEL

Inference optimization is the practice of reducing the cost and latency of running a trained model in production without giving up more quality than a use case can tolerate. That cost is ultimately set by hardware and compute economics, and there are three main levers to pull against it: quantization, which runs the model in lower numerical precision; distillation, which trains a smaller model to imitate a larger one; and speculative decoding, which drafts tokens cheaply and verifies them expensively. Each buys savings, and each pays for those savings differently.

Quantization: Running in Lower Precision

Quantization trades numerical precision for speed and money. Rather than representing the model's weights and activations in 32-bit floating point, it runs them in a lower-precision format such as FP16 or INT8. The numbers are smaller, so the model is smaller, and inference is correspondingly faster and cheaper. It is the cheapest lever to reach for and usually the first one worth trying, since a working implementation is typically a one-time effort measured in days rather than weeks.

The savings are real and the cost is modest. Moving from FP32 to INT8 tends to cut per-query cost by two to four times and latency by roughly half to a third, in exchange for a one to three percent drop in accuracy. A 70-billion-parameter model that costs around $0.60 and two seconds per query at full precision can fall to roughly $0.15 and 600 milliseconds at INT8, at the price of a couple of points of accuracy. The common error is expecting the quality loss to be zero; it is not, and the only way to know its true size for a given workload is to measure it directly on a real test set.

Distillation: Training a Smaller Model

Distillation trades the large model outright for a smaller one trained to mimic its behavior, using the large model's outputs as training data. Where quantization shrinks the representation of a fixed model, distillation produces a genuinely different, smaller model — a 7- or 13-billion-parameter student standing in for a 70- or 405-billion-parameter teacher. The savings are larger and so is the quality hit.

Distilled inference typically runs five to ten times cheaper and faster, but at the cost of a ten to thirty percent quality drop that is inherent to the approach rather than a bug to be tuned away. Because building and training the student is a project of weeks and requires training infrastructure, the economics only work at volume — breakeven arrives somewhere around a hundred million tokens. Before committing to it, it is worth asking whether the traffic even justifies a single small model: if most queries are simple, it is often better to route the easy ones to a smaller, cheaper model and reserve the large model for the hard cases. And distillation can be pushed too far — a 7B student drawn from a 405B teacher can shed thirty to forty percent of the teacher's quality, whereas a 35–70B intermediate usually strikes a better cost-quality balance.

Speculative Decoding: Drafting Fast, Verifying Expensive

Speculative decoding is the one lever that concedes nothing on quality. A small model drafts a run of tokens, and the large model verifies them in a single pass, accepting the ones it agrees with and correcting the rest. Because the large model still has the final word on every token, the output is identical in quality to running the large model alone; what changes is that most of the token generation is done cheaply by the draft model.

In practice this yields two-to-four-times cost savings and two-to-three-times lower latency, with the exact figure depending on how often the large model accepts the draft. At a seventy percent acceptance rate, a query that would cost around $0.60 and two seconds standalone can drop to roughly $0.18 and 600 milliseconds with no measurable quality change. The tradeoff lives elsewhere: it demands a new inference pipeline of some engineering complexity, which is why it fits situations with fixed latency budgets and no tolerance for quality loss better than it fits a quick cost trim.

Choosing Among Them

The right choice follows from the actual bottleneck rather than from whichever technique is most interesting. When cost is the binding constraint, the decision reduces to how much quality can be spared: a one-to-three-percent loss points to quantization, a tolerable ten-to-thirty-percent loss at high volume points to distillation, and an intolerance for any loss points to speculative decoding. When latency is the constraint, quantization paired with batching often suffices without touching quality, speculative decoding covers cases that cannot batch, and only genuinely extreme latency targets justify distillation's quality cost.

The deeper discipline is to optimize the layer that is actually failing. Saving forty percent on inference is worthless if weak retrieval is quietly costing sixty percent in user satisfaction, so it is worth confirming that the retrieval architecture works before spending effort here at all. It is also rarely wise to stack these levers: quantizing and then distilling can compound to a thirty-to-fifty-percent quality loss, and each technique's true numbers only reveal themselves when measured — accuracy, latency, and cost before and after — on a real evaluation set. Inference is one layer of a larger system, and for most products exactly one of these three levers is the one worth pulling.

Open Questions

• As lower-precision formats like FP8 and INT4 mature, how far can quantization be pushed before quality degrades non-linearly, and can that cliff be predicted from a model's architecture rather than discovered empirically per workload?
• When traffic is a mix of easy and hard queries, what is the right division of labor between distillation, routing, and speculative decoding — and can these be composed without the quality losses compounding?
• Every technique here is evaluated against a static test set, but production quality drifts as inputs change; how should teams monitor an optimized model over time so that a tolerable day-one quality loss does not silently widen into an intolerable one?

Part of the knowledge graph at The Best Blog Ever — reference definitions for ideas that matter.