When does model distillation beat quantization for deployment cost and throughput?

Bottom line on distillation vs quantization for deployment cost

Bottom Line: Post-training quantization is the lower-risk default for most existing LLM checkpoints. It avoids a full retraining loop, ships in days rather than weeks, and — according to UniComp (arXiv:2602.09130v3) — quantization provides the best overall trade-off between retained performance and efficiency across hardware-aware evaluations, while distillation yields strong runtime acceleration at high computational cost. Knowledge distillation earns its higher training cost only when sustained inference volume is large enough that the per-token serving savings from a smaller student model accumulate past the upfront compute, evaluation, and engineering spend. The break-even hinges on three variables: sustained request volume (how many tokens per month you serve), latency headroom (whether the quantized teacher already fits your SLA), and quality tolerance (whether the student retains enough task performance). If any one of them breaks the wrong way, quantization wins on every axis: time-to-production, capex, and operational complexity.

UniComp evaluates compressed LLMs across performance, reliability, and efficiency using hardware-aware analysis rather than parameter counts, making it directly applicable to production deployment decisions. Its finding is unambiguous: quantization offers the best overall efficiency trade-off, while distillation produces strong runtime acceleration at high computational cost. That framing maps directly to a deployment-cost decision: quantization is a cost-control tool you apply immediately; distillation is a capital investment that pays back only through sustained serving savings.

What changes the decision: volume, latency, quality, and amortization

The "distillation is smaller, quantization is cheaper" framing misses the real decision inputs. A 7B student model is architecturally smaller than a 70B quantized teacher, but on an H200 with TensorRT-LLM serving INT4-quantized weights, the teacher can still reach very high throughput for the workload under test. The serving math doesn't favor distillation unless the volume-weighted savings over months exceed the training bill. The decision matrix below captures the four inputs that actually shift the break-even.

TensorRT-LLM achieves 11,819 tokens/s on Llama2-13B on a single H200 GPU, with H200 delivering up to 1.9× the throughput of H100 for the same workload. At that throughput level, a quantized teacher can already saturate the hardware; a student model only improves economics if the deployment runs at a scale where GPU-hours are the dominant cost line.

Why throughput gains only matter after you pay back training cost

Distillation produces measurable runtime acceleration — UniComp confirms this — but that acceleration starts generating savings only after the upfront training cost is recovered. Until that point, distillation is a cost liability, not an asset.

Pro Tip: Model throughput from a distilled student is best treated as a savings rate, not a performance number. The student's throughput advantage over the quantized teacher determines how quickly training spend is repaid, not whether deploying the student was the right choice. If your monthly inference spend is materially lower after the student is deployed, you recover the training investment only over a deployment horizon long enough to absorb the upfront work.

As MiniPLM (arXiv:2410.17215v3) states: "Knowledge distillation (KD) is widely used to train small, high-performing student language models (LMs) using large teacher LMs." That framing positions distillation as a training methodology, not an operations shortcut. The economics follow the same logic: training cost is paid upfront, savings accrue over the deployment horizon.

How latency targets narrow the field before cost does

Latency SLAs eliminate distillation as an option before the cost calculation even starts in many deployments. If a quantized teacher already meets the p95 latency target with vLLM 0.6's PagedAttention and continuous batching — which "pages KV cache blocks on demand, treating GPU memory the way an operating system treats RAM, while iteration-level continuous batching keeps the GPU saturated at every forward pass" — there is no latency problem for distillation to solve.

Watch Out: A faster per-token decode rate from a student model does not automatically translate to a lower end-to-end latency under real traffic. If the quantized teacher already sits within your SLA under continuous batching, a student model may still fail to improve the system-level latency picture once request mix and batching behavior are accounted for. Measure end-to-end serving latency under your actual batch distribution, not isolated per-token decode speed.

The practical filter is: quantize first, measure against SLA, then evaluate whether distillation is warranted. Most LLM serving deployments that use TensorRT-LLM with FP8/INT8/INT4 quantization find the latency budget is met without a retraining cycle.

Where quantization wins first in real production budgets

Post-training quantization (PTQ) on an existing checkpoint wins on time-to-production in almost every pilot and low-to-medium volume deployment. It requires no training loop, no distributed infrastructure, and no teacher-student pair. The checkpoint you already have becomes the starting point, not the input to a weeks-long training run.

TensorRT-LLM's quantization support — FP8, INT8, and INT4 including Activation-aware Weight Quantization — covers the full precision range, and a single-GPU run of Falcon-180B using INT4 AWQ reaches up to 800 tokens/second on H200, showing that hardware-specific quantization benchmarks can be very high without requiring a student-training cycle.

Up-front cost profile for post-training quantization

PTQ on an existing checkpoint avoids training compute entirely. The cost reduces to: quantization calibration (a forward pass over a calibration dataset), accuracy validation on your task benchmarks, and serving-stack integration.

Production Note: PTQ's engineering footprint is minimal relative to distillation: no distributed training cluster, no DeepSpeed or NCCL configuration, no checkpoint management across training runs, and no student-evaluation suite to maintain. Quantization does require calibration data and post-quantization accuracy validation before production release, but these are bounded operations that still depend on the model, hardware, and serving stack. "Choosing the right quantization method depends on batch size, accuracy requirements, and target hardware," per NVIDIA's TensorRT-LLM guidance, which means FP8 should be the first option evaluated — it provides the best balance of performance improvement with minimal accuracy degradation.

When quantization does not solve the budget problem

Quantization's Achilles heel is accuracy floor and throughput ceiling. FP8 is often close to lossless for many workloads, but the actual impact depends on the model family, calibration method, and target task. INT4 can preserve quality on some workloads and regress materially on others, so no universal degradation figure applies across all model families or bit-widths.

Quantization-aware training (QAT) occupies a middle path: better quality retention than PTQ, but it reintroduces a training loop and the infrastructure that comes with it. If QAT is required to meet quality tolerance, the engineering cost differential between QAT and full distillation narrows enough that distillation's throughput advantage deserves a proper cost comparison.

Where distillation earns back its extra training spend

Distillation's ROI case is straightforward in principle: a smaller student model costs less per token to serve, and if the serving savings compound over enough traffic, they eventually exceed the training cost. The challenge is that the training cost for a high-quality student is not small, and the serving savings are only realized at scale.

UniComp explicitly positions distillation as producing "strong runtime acceleration gains at high computational cost" relative to quantization. That characterization maps directly to the investment math: distillation front-loads cost (training compute, evaluation cycles, engineering time) and back-loads returns (lower per-token inference cost over the deployment lifetime).

The table below illustrates qualitative ROI tiers rather than hard thresholds.

The serving cost savings scale with volume; the training cost does not. That asymmetry is the entire economic argument for distillation.

The cost stack for training a student model

Training a 7B student from a 70B teacher involves three separate cost categories that teams routinely underestimate when they focus only on GPU-hours.

Pro Tip: Separate training compute, evaluation cost, and engineering time when estimating distillation spend. Training compute (the GPU-hours for the distillation training run itself) is the largest and most visible line item, but evaluation cost — running the student against quality benchmarks repeatedly across training checkpoints — is often material on top of training. Engineering time (writing and debugging the distillation pipeline, integrating with Hugging Face Trainer or DeepSpeed, managing NCCL communication across nodes, and iterating on hyperparameters) can easily exceed the compute spend in total hours, especially for teams doing distillation for the first time.

Distillation adds training compute beyond checkpoint transformation — it requires running the teacher in inference mode to generate soft targets, running the student in training mode on those targets, and managing the two-model data pipeline.

When a 7B student actually out-returns a 70B teacher

No universal break-even threshold exists for a 7B student versus a 70B teacher — the economics depend entirely on your traffic profile, SLA, quality tolerance, and infrastructure rates. The heuristic below is a decision guide, not a guarantee.

NVIDIA's H200 throughput data illustrate why the teacher's quantized performance matters to this calculation: at 11,819 tokens/s for a 13B model on a single H200, a quantized 70B teacher running at a lower but still high throughput can satisfy many API serving workloads. The 7B student earns its place only when even that level of throughput is too expensive at the traffic volume you're serving.

What UniComp says about compression trade-offs

UniComp (arXiv:2602.09130v3) evaluates compressed models across the three dominant techniques: pruning, quantization, and distillation. Its evaluation framework is relevant here because it measures efficiency on actual hardware, not by parameter count.

As the paper states: "UNICOMP evaluates compressed models along three dimensions: performance, reliability, and efficiency, using a diverse set of capability- and safety-oriented benchmarks together with a hardware-aware efficiency analysis."

UniComp's finding is that quantization offers the best overall trade-off across all three axes, while distillation delivers strong efficiency gains at higher training cost. The reliability axis is particularly important: compression can degrade model behavior beyond raw accuracy, and UniComp treats reliability as a first-class metric rather than an afterthought.

Why hardware-aware efficiency matters more than parameter count

A 7B student and a 7B quantized model have the same parameter count. Their actual serving cost differs based on memory bandwidth, KV-cache footprint, batch saturation behavior, and how well the model maps to the target accelerator's compute primitives. UniComp's hardware-aware framing reflects this reality; parameter count is a proxy that routinely misleads deployment decisions.

Watch Out: Do not use parameter count as a proxy for serving cost. A distilled 7B student model does not automatically outperform a quantized 13B model on your specific hardware stack. KV-cache pressure, memory bandwidth, and batch behavior under vLLM's continuous batching all affect real throughput. A distilled model with fewer parameters may still underperform a quantized larger model on an H100 or H200 accelerator at your target batch size. Measure hardware-aware efficiency — tokens per second per dollar — not parameters, before making the final selection.

PagedAttention in vLLM treats KV cache management as a first-class concern, paging blocks on demand to keep the GPU saturated. That architecture means a smaller model's throughput advantage shrinks when the larger quantized model is already operating near memory bandwidth saturation under continuous batching.

What the paper implies for production selection

UniComp's abstract-level conclusion translates into a deployable heuristic: quantization is the default starting point for any existing checkpoint, and distillation is the path to pursue only when quantization's efficiency ceiling is insufficient for the traffic and latency requirements.

Pro Tip: Use UniComp's three axes — performance, reliability, efficiency — as a structured evaluation checklist before committing to either path. For quantization: measure performance on your task benchmarks at each precision level, check reliability metrics (safety-oriented behavior, output consistency), and then measure hardware-aware throughput on your actual serving hardware. For distillation: apply the same checklist to the student after training. The paper's framing implies that "quantization provides the best overall trade-off between retained performance and efficiency, while distillation yields strong runtime acceleration gains at high computational cost" — which means distillation is justified in LLM serving only when the runtime acceleration is the specific gap that quantization cannot close.

Decision framework for LLM serving teams

The right compression path for a given LLM serving deployment follows directly from three questions: What is the sustained monthly serving volume? Does the quantized checkpoint already meet the latency SLA? And is there sufficient quality tolerance for the compression method under evaluation?

TensorRT-LLM's 2–5× throughput advantage over vLLM on H100s means that the serving stack itself is a multiplier on both quantization and distillation gains. Teams should select the serving stack before finalizing compression strategy, since the throughput delta between stacks can exceed the delta between compression methods at moderate model sizes.

Choose quantization when time-to-production is the priority

Post-training quantization on an existing checkpoint is the correct default for most LLM serving deployments. It requires no training infrastructure, runs against the existing checkpoint, and produces a production-ready artifact in days.

Bottom Line: For any existing LLM checkpoint that meets quality requirements at FP8 or INT8 precision, apply PTQ first. TensorRT-LLM's quantization pipeline covers the full range from FP8 to INT4 AWQ, and the engineering cost is a fraction of a distillation project. Distillation is not a faster quantization — it is a separate capital investment that requires sustained high-volume serving to justify.

Quality validation is still required after PTQ, but it is a single-pass evaluation, not an iterative training loop. For teams without established distributed training infrastructure, the fully-loaded cost of building that infrastructure for distillation must be added to the student training estimate before any ROI calculation is meaningful.

Choose distillation when volume turns inference into the biggest cost

Distillation becomes the economically correct choice when inference cost dominates the total cost of ownership and the serving savings from a smaller student materially exceed the training investment over the deployment horizon.

Cost estimates are illustrative and infrastructure-dependent. Validate against your actual GPU-hour rates and model-specific throughput measurements.

At 50B+ tokens per month, the payback window is short enough that distillation is the correct investment even accounting for engineering time. At 1B tokens per month, no realistic training cost yields a positive NPV within a 24-month product horizon.

Risks, counterarguments, and failure modes

Both compression paths carry hidden costs that standard ROI calculations miss. Teams that underestimate these costs make the wrong selection and discover the error after the training budget is spent.

Watch Out: Three failure modes consistently appear in production compression projects. First, evaluation drift: a student model or quantized checkpoint that passes initial accuracy benchmarks may degrade on out-of-distribution inputs or after fine-tuning downstream, requiring re-evaluation cycles that were not in the original estimate. Second, quality regression on safety-oriented benchmarks: UniComp treats reliability as a first-class metric because compression disproportionately affects safety and consistency behaviors relative to raw capability benchmarks — a model that passes MMLU may still fail internal safety evals. Third, hardware mismatch: a student model trained on A100s and then deployed to H200s may not match throughput projections if the KV-cache behavior, memory bandwidth profile, or batch saturation point differs between hardware generations. Always benchmark the compressed artifact on the actual target hardware under realistic traffic before committing to a production rollout.

The operational complexity gap between distillation and quantization is also a real risk. TensorRT-LLM "requires more configuration expertise and longer optimization cycles than user-friendly alternatives like vLLM," per NVIDIA's documentation. For distillation, the equivalent risk is a training pipeline that depends on NCCL stability across multi-node DeepSpeed runs — a common source of silent failures that extend training timelines and inflate costs beyond the initial estimate.

Counterargument worth taking seriously: quantization-aware training (QAT) can close most of PTQ's quality gap while remaining cheaper than full distillation. Teams that find PTQ quality insufficient should evaluate QAT before committing to distillation — QAT reintroduces a training loop but does not require a teacher-student architecture, keeping the pipeline simpler and the evaluation scope narrower.

FAQ on distillation vs quantization economics

Sources and references

• UniComp: A Unified Evaluation of Large Language Model Compression (arXiv:2602.09130v3) — primary research source; hardware-aware evaluation of quantization, distillation, and pruning across performance, reliability, and efficiency axes.
• NVIDIA TensorRT-LLM H200 Launch Blog — verified throughput benchmarks including 11,819 tokens/s on Llama2-13B on H200; quantization mode documentation.
• RunPod vLLM PagedAttention and Continuous Batching Guide — technical reference for vLLM 0.6 serving architecture and KV-cache management behavior.

Keywords: UniComp (arXiv 2602.09130v3), post-training quantization (PTQ), quantization-aware training (QAT), knowledge distillation, Llama 3.1 70B, 7B student model, vLLM 0.6, TensorRT-LLM, H100 GPU, H200 GPU, PagedAttention, continuous batching, Hugging Face Trainer, DeepSpeed, NCCL