ModelRefs / Self-Attention & Multi-Head Attention — Tutorial
Self-Attention & Multi-Head Attention — Tutorial
The core Transformer operation — how positions attend to each other and why multiple heads help. Covers Self-attention: every position talks to every other.
Overview
The core Transformer operation — how positions attend to each other and why multiple heads help
Level: Advanced. Estimated reading time: 35 minutes.
Self-attention: every position talks to every other
In self-attention, the Q, K, and V matrices are all derived from the same input sequence X via learned linear projections: Q = XW_Q, K = XW_K, V = XW_V.
This means each position can directly attend to every other position — no distance limit. The word "bank" in "river bank" can directly attend to "river" to resolve its meaning, regardless of how many words separate them. Compare to RNNs, where information must pass through intermediate hidden states for each step.
The attention weight matrix (seq_len × seq_len) shows exactly which positions attend to which. Visualising attention heads often reveals interpretable patterns: some heads track syntactic relationships, others track coreference, others track positional proximity.
For a decoder generating text (causal self-attention), a mask sets all entries above the diagonal to -∞ before softmax, so position t can only attend to positions 0…t (no peeking at future tokens).
Multi-head attention
A single attention head computes one weighted combination of values. Multi-head attention runs h parallel attention operations with different learned projections:
head_i = Attention(XW_Qi, XW_Ki, XW_Vi)
Then concatenates and projects: MultiHead(X) = Concat(head_1, ..., head_h) · W_O
Why multiple heads? Different heads can learn to attend to different aspects simultaneously. In BERT, some heads track subject-verb agreement, others track pronoun-antecedent links. A single head must trade off between these.
The dimension split: if d_model = 512 and h = 8 heads, each head uses d_k = 64. Total compute is the same as one attention with d_k = 512, but the multi-head version captures richer relationships.
Standard configuration: d_model = 512, h = 8 (BERT-base), or d_model = 1024, h = 16 (BERT-large).
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Self-Attention & Multi-Head Attention — Tutorial.