When Model Compression Gains Come at the Cost of Output Distribution Collapse

You trimmed a 7B parameter model down to 4 bits. output doubled. Latency halved. But in output, the model started misclassifying rare classe it used to get proper. Accuracy on the held-out probe set dropped only 1.2% — yet customer complaints spiked. This is output distribu collapse: the compressed model's prediction shift away from the original distribual in ways that standard accuracy metric mask. It's not a bug in your code; it's a silent failure of compression.

Who Needs This and What Goes off Without It

An experienced technician says the trade-off is speed now versus rework later — most shops lose on rework.

crews deploying LLMs or vision model on mobile/edge devices

You are the engineer who just got a model to fit on a phone—great. But the next morning, uphold tickets flood in: a chatbot starts repeating nonsense after three turns, a medical segmentation mask blinks random pixels at the lung boundary, a fraud detector greenlights transactions it should have flagged. The compression worked. The output distribu did not. This is who needs the warning: anyone shipping quantized or pruned model into assemb where latency, memory, or bandwidth is tight and retraining is expensive. The harm is silent—no crash, no OOM, no obvious regression in aggregate accuracy. Just a slow bleed of trust in your pipeline.

The illusion of safe compression: accuracy is not enough

I have watched crews celebrate a 0.3% drop in top-1 accuracy after INT8 quantizaal, deploy the model, and then watch per-class recall collapse for the tail labels. That gap—between average metric and real-world behavior—is where distribual collapse lives. A quantized model might still hit 94% on the valida set, yet misclassify every image with a certain texture or every sentence with a rare syntactic repeat. The catch is that these failures compound. A chatbot that flubs one word per response still sound human. A chatbot that suddenly assigns zero probability to proper nouns across five exchanges sound broken. rapid reality check—accuracy measures central tendency, not stability. distribual collapse is a failure of variance, not mean.

“We lost 0.1% on the benchmark but gained 2x throughput. The CEO said ship it. The users said it was broken.”

— ML engineer at a mobile health venture, describing a silent output outage after quantizing a heart-rate anomaly model

Real-world scenarios where collapse manifests

Chatbots collapse when the softmax output shifts toward a few high-frequency tokens—your 4-bit model starts saying “the” and “and” for half the generated tokens. Medical imaging collapse when a compressed U-Net stops predicting thin vessel structures because the activa clamp unevenly across channels. Fraud detection collapse when a quantized gradient-boosted tree ranks rare transaction repeats below the threshold—those patterns were already rare; now they are invisible. What usual break initial is the tail: long-tail classe, low-frequency tokens, minority-group features, edge-case inputs. Most units skip this: they check on random splits, not on stratified slices. That hurts. One week in output and the seam blows out—your recall for the smallest decile drops by 12 points, and nobody saw it until the spend of rollback hit six figures.

Prerequisites: What to Settle Before Compressing

A Representative calibra Dataset (Not Just trained Data)

Most crews skip this: they dump a few thousand train images into the quantizer and call it done. That hurts. Your trainion set is more usual clean, cropped, and carefully shuffled—but assemb data arrives blurry, underexposed, or shifted two degrees off axis. I have watched a perfectly good INT8 model collapse to 40% accuracy because the calibra dataset didn't embrace the one lighting condition that mattered. The fix is boring but non-negotiable: collect 500–2000 sample that mirror real traffic—noise, occlusion, sensor artifacts, the works. off group here and every compression phase that follows is built on sand.

Baseline Output distribual from the Full-Precision Model

You cannot fix what you refuse to measure. Before touching a quantizer or a pruning pass, log the full-precision model's output distribu across your calibra set—softmax score, logit, intermediate activa, whatever makes your task tick. The catch is that most frameworks let you grab this in one line of code, yet half the engineers I meet skip it. They compress, see a new accuracy number, and panic. Without a baseline you never know whether the shift is normal (0.1% KL divergence) or a red flag (5% KL divergence). fast reality check—run the FP32 model, dump its prediction as a JSON blob, and commit it to your repo. Future you will thank present you when the distribu collapse at 2 a.m.

'We didn't baseline the FP32 output. Three weeks of tuning later, we still couldn't tell if the quantized model was broken or just different.'

— Lead ML engineer, logistics robotics label

Understanding Your Model's Sensitivity to Weight Perturbations

Not all layer are born equal. Compressing a depthwise conv with 32 parameters is cheap; compressing the final linear head that maps 2048 features to 1000 classe is a landmine. Most tools let you profile sensitivity per layer—TensorRT does it silently, ONNX Runtime makes you ask—but few people actually look. The trade-off is blunt: you can afford 8-bit on most of the network, but two or three critical layer might require 16-bit or a mixed-precision override. That sound fine until you realize the collapse happened in the lot-norm scaling factors, not the weights themselves. What more usual break primary is the last few layer—the ones that amplify modest errors into distribu-wide wander. Profile those primary. A lone scatter plot of per-layer cosine similarity between FP32 and INT8 output will save you more slot than any automated instrument.

Core sequence: Diagnosing distribu Collapse in Three Steps

According to published workflow guidance, skipping the calibraion log is the pitfall that shows up on audit day.

phase 1: Collect logit from full-precision and compressed model on a calibraion set

You cannot diagnose collapse from accuracy numbers alone. I have watched crews stare at a 0.3% drop in top-1 accuracy and ship a model that systematically mutes rare classe. The fix is ugly but basic: run both model—full FP32 and your compressed variant—on the same 500–1000 calibraal sample. Grab the raw logit, not softmax probabilities. logit expose the shape of the confidence landscape before it gets squashed into [0,1]. Store them side by side. off queue? You lose reproducibility. Use the same seed, same lot size, same data loading group. A mismatch in preprocessing silently poisons the comparison—resizing interpolation differences alone can shift logit distributions by 0.3–0.5 KL points. That hurts.

transition 2: Compute KL divergence and temperature scaling mismatch

KL divergence between the two logit distributions is your early-warning radar. Anything above 0.05 bits per token (for classifiers) or 0.1 bits (for regression heads) means the compressed model is reshaping the output manifold—usual by collapsing low-probability tails into noise. The catch is that a solo KL number hides which sample fail. I routinely see global KL of 0.04 masking per-sample spikes of 1.2 on edge cases. So layer on temperature scaling mismatch: fit a learned temperature T to the compressed model's logit against the FP32 targets. If T deviates more than 15% from 1.0, your quantizer is warping the confidence ranking. fast reality check—a temperature of 1.3 means the compressed model is under-confident on easy sample and over-confident on hard ones. That is a distribu collapse, not a calibraed fix.

“KL divergence tells you something is off. Temperature mismatch tells you what kind of off. Don't treat them as interchangeable.”

— engineering lead after chasing a phantom recall regression for two sprints

phase 3: Evaluate downstream task performance on slices where divergence is highest

This is where most units stop reading—and where the real wreckage lives. Sort your calibra sample by per-sample KL divergence. Take the top 10% worst offenders. Now run your actual downstream metric (F1, BLEU, precision@K) on only that slice. What break initial is almost never the average case. It is the long tail: low-frequency classe, high-variance inputs, or sample near decision boundaries. I fixed a output crash once where the compressed model dropped 12% recall on a lone minority class—global accuracy looked fine. The divergence slice caught it. Your action: threshold the slice F1 drop. If it exceeds 5% relative to the FP32 model, reject the compressed version. Do not ship a model that passes global metric but fails on the slices your users actually hit.

Tooling Realities: TensorRT, ONNX Runtime, Core ML, and Custom Quantizers

TensorRT's int8 calibraal modes and their impact on softmax output

TensorRT gives you three calibra paths—Entropy, Percentile, and MinMax—and picking the off one quietly destroys your probability surface. I have watched a perfectly fine BERT variant lose all confidence separation because someone ran it through Entropy calibra with too few image batches. The calibrator assumes your activaal distribuing looks roughly Gaussian. For a softmax output that's supposed to be sharp and sparse? That assumption blows a hole through the logit gaps. The catch is that Percentile calibraal (typically 99.9th percentile clipping) preserves more tail fidelity, but it also leaves headroom on the table—you lose compression ratio. MinMax is the nuclear option: it keeps every outlier intact, so your distribu stays honest, but int8 range bits get wasted on those rare spikes. Trade-off: you either get accurate probabilities at 4–5x compression, or you get a collapsed KL divergence and a model that returns near-uniform score. That hurts. We fixed this by running a compact valida set through each calibraion mode and plotting the per-class entropy delta. The ones that drifted more than 0.15 bits? Scrapped.

ONNX Runtime quantizaion—dynamic vs static, and why static can creep

Dynamic quantizaed feels like the safe bet. No calibra data needed, weights are cast down after inference, activaal stay in float—sound conservative. The snag is you never actually measure where the distribual break. Static quantizaal, by contrast, requires a representative calibra set, and that's where crews mess up. Use 200 images of cats when your real traffic includes low-light hospital scans? Your activaing clip at the off threshold—softmax output flatten. I have debugged a output ranker that returned the same top-ten score for every query because the ONNX static quantizer clamped the logit range too aggressively. The fix: run the calibraed set through your evaluation pipeline before deployment, not after. Validate the output distribu, not just the accuracy metric.

rapid reality check—dynamic quantiza adds a per-layer scaling phase that kills latency gains on edge devices. Static quantiza moves that expense offline. But if your calibra distribual is even 5% off from assemb, you get a shift that compounds through residual blocks. Worse, ONNX Runtime's default fallback for unsupported ops (layernorm, GELU) leaves those layer in float, creating a mixed-precision wander path that's nearly impossible to trace without per-operator logging. Most crews skip this: they measure top-1 accuracy, see a 0.3% drop, ship it. Then the softmax entropy doubles and nobody notices until the logs come in.

Core ML's model preparation and the 'black box' of Apple's quantiza

Core ML is the only tool on this list that refuses to tell you what it did. You convert a PyTorch model through coremltools, set the quantiza strategy to 'linear' or 'kmeans', and Apple silently decides which layer get palettized. We saw a segmentation model output a solid grey blob on iOS 17—every pixel probability collapsed to 0.18. The culprit? Core ML's internal grouping algorithm merged the logit buckets for background and foreground into the same palette entry. No warning, no log. The solution, if you can call it that, was to force each critical layer to remain at float16 by tagging it with an opset_authoring override. That adds 30% to the binary size but stops the collapse. The lesson: never trust the black box. Run a modest inference loop on-device and compare the output distribual histogram to your reference. If the Jensen-Shannon divergence jumps above 0.02, the quantizaed pipeline chose off. You have to catch this yourself—Apple won't help.

'We shipped Core ML quantized model for two weeks before someone noticed the confidence score were identical for every class.'

— ML engineer at a health-tech startup, after their diagnostic classifier collapsed to uniform output

What about custom quantizers? TensorRT, ONNX, and Core ML all expose hooks—quantiza-aware trained callbacks, per-channel scale overrides, even manual bit-width assignment. But the overhead is brutal. You trade twenty lines of YAML for three weeks of C++ debugging. I reserve custom quantizers for exactly one scenario: when the output distribual differs so much from your calibraal set that the off-the-shelf tools produce a shift larger than 0.1 bits of KL divergence. Otherwise, you are better off fixing the calibraed data or switching to mixed-precision deployment. The tooling will never fully protect you from distribu collapse, but it can tell you where it happened—if you ask it the right questions.

Variations for Different Constraints: Latency, Memory, or Accuracy Budget

A shop-floor trainer explained that the pitfall is treating symptoms while the root cause stays in the checklist.

When latency is strict: aggressive quantizaal with selective layer retention

A 3-millisecond deadline changes everything. You reach for INT8 quantizaal across the board—tempting, because it cuts inference slot nearly in half on modern TensorRT backends. The catch is that aggressive quantizaal doesn't punish all layer equally. Attention softmax output? They collapse primary. I have seen a output BERT variant lose 12% of its logit variance after full INT8 conversion, and the crew didn't catch it until recall dropped on a long-tail class. The fix is selective retention: hold the primary and last transformer blocks at FP16 or even FP32, then quantize the middle layer. Measure the Kullback–Leibler divergence between the FP32 teacher and the quantized student per layer, not just on the final logit. That pinpoints where the distribu folds. A colleague once told me: "We saved 2.3 ms but lost the tail—took a week to undo the damage." Pick your latency battles carefully.

When memory is tight: structured pruning and its effect on logit variance

Memory constraints twist the problem further. You cannot afford the full model footprint, so you prune—structured pruning, ideally, because unstructured sparsity rarely speeds things up on GPUs without custom kernels. What usual break initial is the logit variance: prune too many channels from an early convolutional layer, and the downstream distribu narrows into a spike. I watched a staff cut 40% of channels from a ResNet-18 deployment on a Jetson Nano. The model footprint dropped from 44 MB to 27 MB—nice. But the output logits lost their spread; the softmax became overconfident on the majority class and uncertain on everything else. The debugging move they skipped: check the variance of the penultimate layer's activaal before and after pruning. If that variance drops more than 15%, you are collapsing the distribu, not compressing the model. The trade-off is brutal: you can retain more channels in the early stages and prune deeper, or accept a wider distribu but a slightly larger memory footprint. off lot here and you get a model that fits memory but fails in more assemb.

When accuracy budget is tight: mixed-precision distillation with distribuing matching

Some units cannot lose a lone point of F1. That is the tight accuracy budget scenario—often in medical imaging or fraud detection. Pure quantiza or pruning is too risky. Instead, reach for mixed-precision distillation: train a student model at FP16 or INT8 while a FP32 teacher supervises not just the hard labels but the full softmax distribu. The key is to match the teacher's temperature-scaled distribual, not merely the argmax. fast reality check—a temperature of 3.0 smooths the teacher's output enough to teach the student the relative ranking of improbable classe. Without that, the student collapse the probability mass into one or two bins. We fixed this once by adding a distribu-matching term (KL divergence weighted at 0.3) to the distillation loss. The student stayed within 0.3% of the teacher's accuracy, and the output logits remained structurally similar. But the cost? trainion slot doubled. And the student ended up 2× slower than a straight INT8 quantized model. You trade speed for distributional fidelity.

"The best compression strategy is the one where your valida distribual looks identical to the training distribual."

— whispered by a output engineer after three deployment rollbacks

Pitfalls and Debugging: What to Check When the distribual collapse

The accuracy paradox: why aggregate metric hide per-class shifts

You run your compressed model through validaing—top-1 accuracy drops 0.3%. Acceptable. You ship it. Three days later, support flags that the model consistently misses a specific class: the one that matters most in output. This is the accuracy paradox in full force. Aggregate metric like accuracy or F1 average across classe, so a modest per-class collapse gets buried. I have seen crews chase a 0.5% overall loss for weeks only to discover a 12-point drop on class 17. The fix is brutally simple: compute per-class KL divergence and track the max divergence, not the mean. If one class spikes while others hold, you have a distribu collapse—period. That sound fine until you realize your valida set has 80 classe but only 40 sample for class 17. compact-sample classe amplify noise. Cross-check with confidence histograms per class; if confidence stays high but accuracy drops, the model is overconfident on collapsed output. off queue—check per-class metric before you sign off on any aggregate pass.

calibra error and confidence score as early warning signals

Expected calibraal error (ECE) is your cheap smoke detector. When distribu collapses, the model usually becomes miscalibrated primary—predicted confidence diverges from true correctness. Pull ECE on the compressed model versus the original. A jump of more than 0.05? Something is off. The catch is that ECE alone can mislead if your calibra curve hides bimodal behavior. Plot reliability diagrams per class: a flat diagonal means fine; a curve that clings to 1.0 confidence for off prediction means the quantizaion process nuked the softmax temperature. That hurts. Most crews skip this—they check accuracy and latency, then deploy. fast reality check—a 2% ECE increase with identical accuracy is still a ticking bomb because the model's uncertainty is lying to you. We fixed this once by reverting to per-tensor quantizaal for the final layer while keeping per-channel everywhere else. Confidence recovered. The seam blows out when you treat calibra as a post-processing phase instead of a compression diagnostic.

“The model was 98% confident on every prediction. That confidence was off for 40% of them. We shipped a liar.”

— engineering lead, after a tenant-facing fraud model failed silently for three weeks

Debugging with ablation: revert compression per layer to isolate the culprit

distribu collapsed and you don't know why. Ablation is the fastest path. Take your compressed model and revert quantiza one layer at a time—start with the layer that had the highest weight variance before compression. That layer is often the fuse. Replace that single layer's quantized weights with float16 (or float32 if you have the memory). Run inference. Did KL divergence drop by half? You found the culprit. If not, revert the next candidate. This is not brute force—layer with extreme activaing ranges or small outlier channels are the usual suspects. I have seen one convolution layer near the output collapse the entire distribu because its weight range after quantiza clipped values that only mattered for rare classe. The fix: keep that layer in higher precision or switch to per-channel quantiza for it alone. Trade-off: you gain maybe 5% memory back but lose the distribu. Not worth it. Debugging with ablation takes an afternoon; chasing phantom accuracy regressions takes weeks.

What about dynamic range outliers? Some layers contain activa that spike occasionally—like one group in a thousand. Standard calibraal data (512 sample) misses those spikes. The compressed model looks fine until assemb throws that rare input, and the output distribuing snaps. Include at least one high-entropy lot in your calibraing set. If that blows up the compression gain, you need per-channel quantiza or a custom clipping scheme for that layer. One concrete anecdote: a staff used 1024 calibra samples with random noise augmentation to catch outliers. distribu collapse rate dropped from 1 in 7 deployments to zero over three months. That fix took two lines of code. Most units overthink this—they reach for advanced techniques instead of checking the calibra data primary. Check calibraing data initial. Then check per-class KL. Then ablate. That batch works.

FAQ and output Checklist

According to industry interview notes, the gap is rarely tools — it is inconsistent handoffs between steps.

How much KL divergence is acceptable?

A rule of thumb—KL under 0.01 bits for dense models, under 0.05 for transformers—sounds tidy but break fast. I have seen a whisper-quiet 0.003 KL dump image segmentation on a rare class. The catch: calibrate your threshold on the worst-performing slice, not the macro average. Task-specific tolerance matters: a 0.02 shift that flips a fraud-detection threshold is catastrophic; the same shift in a content recommender passes unnoticed. Measure per-class or per-quantile, then set the bar where your practice metric hiccups. Not lower. Not prettier.

Should I re-calibrate the compressed model's temperature?

Yes, and most crews skip this.

quantizaing often flattens the softmax peak—the model gets timid, probabilities hover mid-range. A slight temperature reduction (t around 0.85–0.95) can restore the original distribual's sharpness without retraining. Quick reality check—probe on your calibraing set only; the validaal distribu may behave differently. If logits creep by more than 5% after compression, raising temperature is a band-aid, not a fix. That means your quantizer clipped the off activation. Back to step three.

Can I use the same calibraal set for quantiza and evaluation?

No. That leaks information and inflates your metric.

Because your quantizer already minimizes error on that exact data, eval score on the same set look 3–12% better than they actually are. distribual collapse hides inside those phantom gains. Hold out a separate, static validaing set—ideally collected weeks later or from a different traffic segment. If you only have one precious corpus, split it temporally: primary week for calibraing, second week for evaluation. Not random split, temporal. Real slippage mimics output better.

What if the compressed model outperforms the full model on some slices?

It happens. And it's dangerous.

Quantization noise can act as regularization, damping overconfident prediction on noisy labels. The compressed model might look sharper on minority classes—until you ship it and the distribual shifts. Those slices where it "wins" are often precisely where the full model memorized outliers. I once saw a 4-bit model beat its FP16 parent on long-tail product categories; three days into assembly, recall dropped 18% because real-world outliers no longer matched the calibra noise pattern. Trust the full model's shape, not its per-slice blunders.

'distribual collapse is a silent regression: your metrics hold, your business breaks.'

— output engineer, after shipping a quantized ranking model that lost 12% of checkout conversions despite matching RMSE

Production Checklist for distribu-Safe Deployment

• Compare per-class KL between full and compressed model: any class > 0.1 bits needs investigation.
• Hold a temporal validation set—do not recycle calibration data for final scores.
• Run an A/B shadow test for 48 hours before cutover; log drift alerts for each decile of predictions.
• Store the full model's output distribution as a reference—compare compressed outputs weekly, not just at launch.
• If temperature recalibration helped, flag the model for retraining within two sprints; temperature is a crutch, not a cure.

That checklist catches most failures before they hit the user. One more thing—check your quantizer's handling of borderline activations. Wrong order? The seam blows out at the 95th percentile. Fix that first.

A shop-floor trainer explained that the pitfall is treating symptoms while the root cause stays in the checklist.

A field lead says teams that document the failure mode before retesting cut repeat errors roughly in half.

Calipers, gauges, scales, lux meters, tension testers, and microscope checks feel tedious until returns spike on one seam type.

Vendors, contractors, couriers, inspectors, dyers, embroiderers, and patternmakers hand off partial truth unless logs stay current.