Quantization in Large Language Models

Quantization in Large Language Models

Every large language model ships with billions of parameters stored as numbers, and the precision you choose to store and compute those numbers in has a direct, almost linear effect on memory footprint, inference latency, and serving cost. Quantization is the set of techniques for reducing that numeric precision, from 16-bit or 32-bit floating point down to 8-bit, 4-bit, or even lower, while trying to preserve as much of the model's original behavior as possible.

This post covers how quantization actually works at the arithmetic level, why naive quantization breaks large models in specific and predictable ways, and how the current generation of methods (LLM.int8, GPTQ, AWQ, SmoothQuant, GGUF-style k-quants, and QLoRA) each solve a different piece of that problem.

Why Quantization Matters for LLMs Specifically

A 70 billion parameter model stored in 16-bit floating point needs 140 GB just to hold the weights, before accounting for activations, the KV cache, or optimizer state during training. That alone exceeds the memory of a single high-end GPU. Quantizing to 8-bit halves that to 70 GB, and 4-bit quantization brings it down to roughly 35 GB, small enough to fit on a single consumer-grade or prosumer GPU.

But the motivation isn't only memory capacity. Inference on modern accelerators is frequently memory-bandwidth bound rather than compute bound, especially during autoregressive decoding where you generate one token at a time and each step reads the entire weight matrix to produce a single vector of activations. Lower-precision weights mean fewer bytes to move from memory to compute units per token, which translates directly into higher tokens-per-second throughput, even when the arithmetic itself is upcast back to a higher precision before the actual matrix multiply.

Numeric Formats: What "Precision" Actually Means

Floating point

A floating point number is represented as sign, exponent, and mantissa (fraction) bits. The exponent gives it a wide dynamic range (it can represent both very large and very small numbers), while the mantissa determines precision within that range.

BF16 became the default training format for large models because it keeps FP32's exponent range (so training doesn't overflow or underflow as easily) while sacrificing mantissa precision, which turns out to matter less for the noisy, redundant computations inside deep networks.

Integer quantization

Most aggressive LLM quantization methods don't use narrower floating point formats, they map floating point values onto a small integer grid (commonly INT8 or INT4) and store a separate floating point scale (and sometimes a zero-point) per tensor, per channel, or per small group of values. Reconstructing an approximate float value at compute time is a single multiply-add:

real_value ≈ scale * (int_value - zero_point)

For symmetric quantization, the zero-point is fixed at zero, and the range is chosen to be symmetric around zero (for example, -127 to 127 for INT8). This is simpler and cheaper but wastes representational range if the underlying values are not centered at zero.

For asymmetric quantization, the zero-point is a learned or computed offset, letting the integer range map to an arbitrary real-valued interval (for example, weights that are mostly positive). This uses the available integer levels more efficiently at the cost of an extra subtraction during dequantization.

def quantize_symmetric_int8(x):
    scale = x.abs().max() / 127.0
    q = torch.round(x / scale).clamp(-127, 127).to(torch.int8)
    return q, scale

def dequantize(q, scale, zero_point=0):
    return scale * (q.to(torch.float32) - zero_point)

The core trade-off is always the same: fewer bits means a coarser grid, which means larger rounding error per value, but the scale and zero-point are exactly what let you recover something close to the original distribution instead of just truncating it.

Granularity: Per-Tensor, Per-Channel, Per-Group

The granularity at which you compute the scale factor is one of the biggest levers on quantization quality, often more important than the bit width itself.

• Per-tensor quantization uses a single scale for an entire weight matrix. It's cheap and simple, but a single outlier value anywhere in the tensor forces the scale to be large, which crushes the resolution available to every other, more typical value in that tensor.
• Per-channel (per-column or per-row) quantization assigns a separate scale to each output channel of a weight matrix. Since different channels in a neural network layer often have very different natural magnitudes, this dramatically reduces the damage any single channel's outliers do to the rest.
• Per-group (block-wise) quantization goes further, splitting each channel into small groups (commonly 32, 64, or 128 values) and assigning each group its own scale. This is the approach used by GPTQ, AWQ, and GGUF's k-quant formats, and it is what makes 4-bit weight quantization viable at all: an entire tensor's worth of outliers can no longer distort the quantization of unrelated groups of weights.

The cost of finer granularity is metadata overhead: every additional scale factor takes memory too. A group size of 128 for INT4 weights adds roughly one 16-bit scale (and sometimes a zero-point) per 128 packed 4-bit values, which is a small overhead (about 1 bit per weight) relative to the savings from going from 16 bits to 4 bits.

Post-Training Quantization vs. Quantization-Aware Training

Post-training quantization (PTQ) takes an already-trained model and quantizes its weights (and sometimes activations) afterward, typically using a small calibration dataset to compute good scale factors and, in more advanced methods, to correct for the quantization error introduced. PTQ is fast (minutes to a few hours for large models) and doesn't require access to the original training pipeline or dataset, which is why it dominates in practice for open-weight model quantization.

Quantization-aware training (QAT) simulates quantization during the forward pass of training or fine-tuning (using "fake quantization," rounding then immediately dequantizing so gradients can still flow), letting the model's weights adapt to compensate for quantization error. QAT generally produces better accuracy at very low bit widths, but it requires a full training or fine-tuning run, access to representative training data, and significantly more compute, which is why it's mostly reserved for cases where PTQ accuracy loss is unacceptable.

Because gradients cannot flow through a discrete rounding operation, QAT relies on the straight-through estimator: during the backward pass, the rounding operation is treated as if it were the identity function, so gradients pass through unchanged even though the forward pass used the rounded value. This is an approximation, not a mathematically exact gradient, but it works well enough in practice to be the standard trick across nearly all QAT implementations.

The Outlier Problem: Why Naive LLM Quantization Fails

Naive INT8 quantization works reasonably well for smaller models and for convolutional networks, but it degrades badly on large transformer-based language models once you cross roughly the 6 to 7 billion parameter range. The reason, identified in the LLM.int8 paper (Dettmers et al., 2022), is that large transformers develop a small number of emergent outlier features: specific hidden dimensions where activation magnitudes are 20 to 100 times larger than the typical value, appearing consistently across many tokens and layers once a model is large enough.

These outliers are not noise, they carry disproportionate weight in the model's actual function (the model relies on some of them for tasks like maintaining attention sink behavior), so simply clipping or rounding them away meaningfully degrades output quality. But because per-tensor and even per-channel scales are set by the maximum absolute value, a handful of huge outlier activations force the quantization scale up for an entire tensor, crushing the precision available to every normal-magnitude value around them.

LLM.int8: mixed-precision decomposition

LLM.int8 addresses this directly: it identifies the small number of outlier feature dimensions in the activations (typically well under 1% of dimensions) at runtime, and routes just those dimensions through a regular FP16 matrix multiply, while the remaining, non-outlier dimensions are quantized to INT8 and multiplied there. The two partial results are then added together. This mixed-precision decomposition means the expensive, high-precision path only ever touches a tiny slice of the computation, so most of the memory and bandwidth savings of INT8 are preserved while outlier-sensitive dimensions are computed at full precision.

SmoothQuant: shifting difficulty from activations to weights

SmoothQuant (Xiao et al., 2022) takes a different approach: instead of special-casing outliers at runtime, it mathematically migrates the quantization difficulty from activations to weights before quantizing either. Activation outliers are the hard part to quantize (they're input-dependent and can't be precomputed), while weights are comparatively easy to quantize since they're static and already well-behaved.

SmoothQuant applies a per-channel scaling factor that divides activations by a smoothing factor and multiplies the corresponding weight channel by the same factor, which is mathematically a no-op for the matrix multiply as a whole, but shifts magnitude from the hard-to-quantize activations onto the easy-to-quantize weights:

Y = (X / s) * (s * W) = X * W

The smoothing factor for each channel is chosen based on calibration data to balance how "quantization-friendly" the activation and weight distributions become, effectively trading a bit of weight quantization difficulty for a large reduction in activation quantization difficulty. This lets both weights and activations be quantized to INT8 with much smaller accuracy loss than naive per-tensor INT8 quantization, without needing LLM.int8's runtime mixed-precision path.

Weight-Only Quantization at 4 Bits: GPTQ and AWQ

For most practical local and single-GPU inference workloads, the dominant approach today is weight-only quantization: quantize weights to 4 bits (sometimes 3 or even lower), but keep activations in FP16 or BF16. This sidesteps the activation outlier problem entirely, since activations are never quantized, while still delivering most of the memory savings, because weights, not activations, dominate a large model's static memory footprint.

GPTQ

GPTQ (Frantar et al., 2022) frames weight quantization as a layer-wise reconstruction problem: for each layer, find quantized weights that minimize the squared error between the original layer's output and the quantized layer's output on a calibration dataset, rather than just rounding each weight independently. It processes weights column by column, and after quantizing each column, it updates the remaining, not-yet-quantized columns to compensate for the error just introduced, using an approximation derived from the layer's Hessian (second-order error information). This sequential error-compensation is what allows GPTQ to push weights down to 4 or even 3 bits while keeping downstream task accuracy close to the original FP16 model, far closer than naive rounding would achieve at the same bit width.

Because GPTQ only needs a modest calibration dataset (a few hundred sequences of generic text is typical) and no gradient-based training, it can quantize a 70 billion parameter model in a few GPU-hours rather than requiring a full fine-tuning run.

AWQ (Activation-Aware Weight Quantization)

AWQ (Lin et al., 2023) starts from a different observation: not all weights matter equally, a small fraction of weight channels correspond to consistently large-magnitude activations, and protecting the precision of just those "salient" weight channels preserves accuracy far more effectively than treating every weight as equally important. Rather than solving a per-layer reconstruction problem like GPTQ, AWQ analyzes activation statistics from calibration data to identify which weight channels are salient, then computes a per-channel scaling factor (conceptually similar to SmoothQuant's approach, but applied to protect weight precision under quantization rather than to ease activation quantization) that reduces the relative quantization error specifically on those important channels.

AWQ tends to be faster to run than GPTQ (no iterative per-layer optimization, just statistics collection and a scaling search) and is often competitive with or slightly better in accuracy at very low bit widths, though the two methods are close enough in practice that the choice frequently comes down to tooling and inference-engine support rather than a clear accuracy winner.

GGUF and k-quants: Quantization for CPU and Edge Inference

The llama.cpp ecosystem popularized the GGUF file format alongside a family of quantization schemes generally called k-quants (for example, Q4_K_M, Q5_K_S, Q8_0). These schemes are built around the same per-group scale-and-zero-point idea described earlier, but are specifically engineered for fast dequantization on CPUs and consumer GPUs using integer SIMD instructions, rather than requiring a dedicated CUDA kernel.

The naming convention encodes the bit width and a quality/size tradeoff tier: Q4_0 and Q4_1 are older, simpler per-block schemes; the newer Q4_K variants use two-level scaling (a per-superblock scale refined by per-sub-block scales) to squeeze out better accuracy at the same average bit width. The "_S," "_M," and "_L" suffixes (small, medium, large) indicate different splits of which layers get more or fewer bits, since some layers (commonly attention output projections and certain feed-forward weights) are more sensitive to quantization error than others and are worth spending extra bits on.

This matters for a broader point: not all layers in a transformer are equally quantization-sensitive, and mixed-precision schemes that spend more bits on error-sensitive layers (and fewer bits on error-tolerant ones) consistently beat uniform bit-width assignment at the same average memory budget.

KV Cache Quantization

Weight quantization addresses static memory, but during long-context inference, the key-value cache (the attention keys and values stored for every previous token) can dwarf the size of the weights themselves at large batch sizes or long sequence lengths. KV cache quantization, typically to INT8 or INT4, applies the same scale-and-zero-point mechanics to these activation tensors instead of weights.

This is harder than weight quantization for two reasons: the KV cache is generated dynamically at inference time (there's no offline calibration pass over the actual values you'll see), and it's appended to incrementally, one token at a time, which constrains you to per-token or per-channel quantization schemes that can be computed online rather than requiring a full pass over the data in advance. In practice, per-channel scales (computed once, since a channel's typical magnitude range is relatively stable across tokens) combined with per-token dynamic scaling factors tend to work well, and INT8 KV cache quantization is now common in serving stacks like vLLM for extending effective context length without a proportional memory cost increase.

QLoRA: Quantization Meets Fine-Tuning

QLoRA (Dettmers et al., 2023) combines quantization with parameter-efficient fine-tuning rather than treating them as separate concerns. The base model's weights are quantized to 4 bits using NF4 (4-bit NormalFloat), a data type specifically designed around the empirical observation that pretrained neural network weights are approximately normally distributed. Instead of an evenly spaced integer grid, NF4's quantization levels are placed at the quantiles of a standard normal distribution, so that each of the 16 representable 4-bit values covers an equal amount of probability mass under a typical weight distribution rather than an equal amount of numeric range. This information-theoretically better matches how weight values are actually distributed, reducing quantization error relative to a uniform INT4 grid at no extra storage cost.

During fine-tuning, the quantized base weights are frozen and only small, full-precision LoRA adapter matrices are trained on top, with gradients computed by dequantizing the frozen weights on the fly for the forward and backward pass. QLoRA also introduces double quantization (quantizing the quantization scale factors themselves, since storing one 32-bit scale per small block adds up) and paged optimizer states (using unified memory to avoid out-of-memory crashes during optimizer state spikes), together enabling fine-tuning of a 65 billion parameter model on a single 48 GB GPU, a workload that would otherwise require multiple high-memory GPUs at full precision.

Measuring Quantization Quality

Two complementary signals are used in practice to judge whether a given quantization scheme is "good enough":

• Perplexity on a held-out text corpus (commonly WikiText-2 or a similar dataset) captures how well the quantized model's next-token probability distribution matches the original model's. It's cheap to compute and sensitive to quantization degradation, but doesn't always correlate cleanly with downstream task performance.
• Downstream task benchmarks (multiple-choice reasoning, coding, instruction-following evaluations) more directly reflect real-world usability, but require more compute to evaluate and can mask localized degradation (a model might do fine on average while failing badly on a narrow but important capability, like arithmetic or long-context retrieval).

A model quantized to 8-bit weight-only typically shows near-negligible perplexity increase over the FP16 baseline. At 4-bit with a well-tuned method like GPTQ or AWQ, the degradation is usually small but measurable. Below 4 bits (3-bit, 2-bit), degradation grows quickly and becomes uneven across capabilities, which is why sub-4-bit quantization remains comparatively niche and typically requires QAT or more elaborate mixed-precision schemes to stay usable.

Trade-offs Summary

Practical Guidance

Choosing a quantization strategy comes down to answering three questions: are you serving on GPU or CPU, do you need to fine-tune or only run inference, and how much accuracy loss is acceptable for your use case.

• For GPU inference where minimizing latency and maximizing throughput matter most, weight-only 4-bit methods (GPTQ or AWQ) paired with an inference engine that has fused dequantization kernels (vLLM, TensorRT-LLM, ExLlama) give the best combination of memory savings and speed, since activations stay in FP16 or BF16 and avoid the added complexity of activation quantization.
• For CPU or edge inference, GGUF k-quants are the practical default, since llama.cpp's kernels are specifically built around efficient dequantization using CPU SIMD instructions rather than requiring a GPU.
• For fine-tuning a large model on limited hardware, QLoRA is the standard approach, since it's the only method here designed to keep weights quantized during a training loop rather than only at inference time.
• For maximum safety margin with minimal engineering effort, 8-bit approaches (LLM.int8, SmoothQuant, or even a plain per-channel INT8 scheme) remain a reasonable choice when memory isn't the binding constraint, since accuracy loss at 8-bit is close to negligible across nearly all published evaluations.

Closing Notes on Limitations

• Quantization error compounds with model capability in subtle ways: reasoning-heavy, long-chain-of-thought tasks and precise numeric or code generation tend to be more sensitive to aggressive quantization than general text generation, because small per-step errors can accumulate across a long generation.
• Calibration-based methods (GPTQ, AWQ, SmoothQuant) are only as good as their calibration data; a calibration set that doesn't represent your actual deployment distribution (for example, calibrating on English web text but deploying for code generation) can leave the quantized model worse on your specific workload than the aggregate benchmarks suggest.
• Extremely low bit widths (2-bit, and even some 3-bit schemes) still generally require either QAT or considerably more sophisticated mixed-precision and error-compensation techniques to remain usable, naive PTQ at that range tends to degrade sharply rather than gracefully.
• Hardware support lags algorithms: a quantization scheme is only as fast in practice as the kernels available for it, which is why weight-only 4-bit formats with mature kernel support (GPTQ, AWQ) often see wider production adoption than academically superior but less-optimized alternatives.

The throughline across every method here is the same: quantization is fundamentally a budget allocation problem, deciding where to spend a limited number of bits so that the values and layers most responsible for model behavior keep enough precision, while everything else is compressed as aggressively as possible.

Never. Stop. Exploring.