ModelRefs / GPT & Decoder Models — Tutorial
GPT & Decoder Models — Tutorial
Autoregressive generation, temperature sampling, and the decoder-only architecture that powers modern LLMs. Covers Autoregressive generation.
Overview
Autoregressive generation, temperature sampling, and the decoder-only architecture that powers modern LLMs
Level: Advanced. Estimated reading time: 35 minutes.
Autoregressive generation
GPT (Generative Pretrained Transformer, OpenAI 2018) uses a decoder-only Transformer pretrained on a single task: predict the next token given all previous tokens. This is called language modelling or autoregressive generation.
Pretraining objective: P(x_1, x_2, ..., x_n) = Π P(x_t | x_1, ..., x_{t-1}). Maximise the probability of the next token at every position.
At inference, start with a prompt, sample the next token from the predicted distribution, append it, repeat. This generates text one token at a time.
Scale dramatically improved capabilities: GPT-1 (117M params) → GPT-2 (1.5B) → GPT-3 (175B) → GPT-4 (rumoured ~1T across mixture of experts). Each jump revealed new emergent capabilities — GPT-3 showed few-shot learning (solve tasks from a few examples in the prompt with zero gradient updates).
Sampling strategies
The model outputs logits over the vocabulary. How you convert these to the next token affects the quality and diversity of generated text:
Greedy decoding: always pick the highest-probability token. Deterministic but repetitive and often degenerate.
Temperature sampling: divide logits by temperature T before softmax. T=1: original distribution. T<1: sharper (more confident, less diverse). T>0.7: typical for creative tasks. T→0: approaches greedy.
Top-k sampling: only sample from the k most probable tokens (k=50 is common). Prevents sampling very unlikely tokens.
Top-p (nucleus) sampling: sample from the smallest set of tokens whose cumulative probability exceeds p (p=0.9 typical). Adapts the cutoff to the confidence of the distribution.
In practice: temperature + top-p together. For factual tasks: T=0.1, top-p=0.9. For creative writing: T=0.9, top-p=0.95.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to GPT & Decoder Models — Tutorial.