ModelRefs / Model Quantization & ONNX — Tutorial
Model Quantization & ONNX — Tutorial
INT8, GPTQ, AWQ, and ONNX export — reduce model size 4× and latency 2× with minimal accuracy loss. Covers What quantization does, GPTQ and AWQ for LLMs.
Overview
INT8, GPTQ, AWQ, and ONNX export — reduce model size 4× and latency 2× with minimal accuracy loss
Level: Expert. Estimated reading time: 40 minutes.
What quantization does
Neural network weights are stored as float32 (4 bytes each) by default. A 7B parameter model needs 28GB in fp32. Quantization reduces numerical precision:
fp16 / bf16: 2 bytes per parameter — 14GB for 7B. Standard for GPU inference. Almost no accuracy loss for most models.
INT8: 1 byte per parameter — 7GB for 7B. ~2× faster on hardware with INT8 accelerators. Accuracy loss is 0.5–2% on most benchmarks.
INT4 (GPTQ, AWQ): 0.5 bytes per parameter — 3.5GB for 7B. Fits a 7B model on a consumer GPU (8GB VRAM). Accuracy loss 1–4% depending on task and method.
The core idea: round weights to the nearest representable value in the target precision. The challenge: some weights are more sensitive than others — naive rounding causes large accuracy drops. Advanced methods (GPTQ, AWQ) find optimal rounding that minimises output perturbation.
Dynamic vs static quantization: dynamic quantizes activations at runtime (simpler, slight overhead); static quantizes activations using calibration data (faster at inference, requires calibration step).
GPTQ and AWQ for LLMs
For large language models, two methods dominate:
GPTQ (Frantar et al., 2022): quantizes each layer by solving a small optimisation problem that minimises reconstruction error. Applied layer-by-layer. Requires a calibration dataset (128 samples of the target domain) but no fine-tuning. Works well at 4-bit; degrades at 3-bit for most models.
AWQ (Lin et al., 2023): Activation-aware Weight Quantization. Key insight: not all weights are equally important — weights activated by large input channels matter more. AWQ identifies and protects salient weights (scales them before quantization). Slightly better than GPTQ at 4-bit for most benchmarks.
Practical usage: pre-quantized GPTQ and AWQ models for all major open-source models are available on Hugging Face (look for -GPTQ or -AWQ suffixes). Run them with AutoGPTQ or llm.int8() from bitsandbytes.
Running quantized LLMs: vLLM, llama.cpp, Ollama, and TGI (Text Generation Inference) all support 4-bit quantized models and add paged attention, continuous batching, and speculative decoding for production throughput.
ONNX: exporting models for cross-platform inference
ONNX (Open Neural Network Exchange) is a standard format for ML models. Export once from PyTorch or TensorFlow; run anywhere — CPU, GPU, mobile, browser, edge device.
Why ONNX: PyTorch models need the PyTorch runtime (~2GB). ONNX Runtime is ~50MB and optimised for inference on all platforms including ARM, x86, and Windows.
Optimizations from ONNX Runtime: operator fusion (combine Conv+BatchNorm+ReLU into a single kernel), constant folding, layout optimization. Typically 1.5–3× speedup vs raw PyTorch inference.
ONNX for transformers: the Optimum library from Hugging Face exports BERT/GPT/Whisper/etc. to ONNX in one line and supports INT8 quantization via ONNX Runtime's quantizer. A quantized ONNX BERT runs 4× faster on CPU than the original PyTorch model.
Deployment targets: ONNX Runtime Web (WebAssembly, runs in browser), ONNX Runtime Mobile (iOS/Android), TensorRT (NVIDIA GPU — ONNX→TensorRT gives the highest GPU throughput).
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Model Quantization & ONNX — Tutorial.