ModelRefs / Linear Algebra Essentials — Tutorial
Linear Algebra Essentials — Tutorial
Vectors, matrices, and the operations that power every neural network — explained with NumPy. Covers Vectors: direction and magnitude.
Overview
Vectors, matrices, and the operations that power every neural network — explained with NumPy
Level: Beginner. Estimated reading time: 22 minutes.
Vectors: direction and magnitude
A vector is an ordered list of numbers. In machine learning, a vector usually represents a data point, a word embedding, or a layer's activations.
A 3-dimensional vector [1, 2, 3] can be thought of geometrically as an arrow from the origin to the point (1, 2, 3). Two operations matter most:
Dot product: a · b = Σ(aᵢ × bᵢ). For unit vectors, the dot product equals cosine similarity — measuring how "aligned" two vectors are. This is how embedding search works.
Magnitude (norm): ||a|| = √(a₁² + a₂² + ... + aₙ²). Normalising a vector divides it by its magnitude to get a unit vector.
Matrices: the core data structure of deep learning
A matrix is a 2D array of numbers with shape (rows, cols). In a neural network, a weight matrix W transforms an input vector x into an output: y = Wx + b.
Key operations: - Matrix multiply: (m×n) @ (n×p) → (m×p). The inner dimensions must match. - Transpose: flip rows and columns. W.T has shape (cols, rows). - Element-wise multiply: same-shape matrices, multiply position-by-position (used in attention masking).
A batch of 32 training examples, each with 784 features (28×28 image), is a (32, 784) matrix. Multiplying by a (784, 128) weight matrix gives a (32, 128) output — all 32 examples processed simultaneously.
Why linear algebra is unavoidable in AI
Every layer of a neural network is a matrix multiplication followed by a non-linearity. Attention in a Transformer computes Q @ K.T to get attention scores — a matrix multiply. Backpropagation passes gradients backward using the chain rule, which is vector-Jacobian products (matrix multiplies).
GPUs are fast precisely because they are designed to run thousands of matrix multiplications in parallel. A modern GPU can do ~300 TFLOPS of FP16 matrix operations — that's 300 trillion multiply-adds per second.
You don't need to implement matrix multiply by hand, but understanding shapes — knowing that (batch, seq_len, d_model) @ (d_model, d_ff) → (batch, seq_len, d_ff) — is essential for debugging model architectures.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Linear Algebra Essentials — Tutorial.