Backpropagation — Tutorial

The chain rule applied to networks

Backpropagation is automatic differentiation applied to a computational graph. It computes ∂L/∂w for every parameter w by applying the chain rule: ∂L/∂w = (∂L/∂a) · (∂a/∂w), where a is the activation in between.

The key insight: gradients flow backward through the network in the same graph that activations flowed forward. Each layer's backward pass computes the gradient of the loss with respect to its inputs and parameters, then passes the input gradient to the previous layer.

Forward and backward pass

Forward pass: compute the output and loss. Cache all intermediate activations — backprop needs them.

Backward pass: start from the loss gradient (∂L/∂L = 1). For each layer in reverse order: 1. Receive gradient from the next layer: ∂L/∂output 2. Compute gradient w.r.t. parameters: ∂L/∂W = ∂L/∂output · ∂output/∂W 3. Compute gradient w.r.t. input: ∂L/∂input = ∂L/∂output · ∂output/∂input 4. Pass ∂L/∂input to the previous layer

PyTorch's autograd does all of this automatically via loss.backward().

You should never implement backprop by hand in production

Understanding backprop conceptually is essential for debugging (why are gradients exploding? why is a layer not learning?). But implementing it manually is error-prone and unnecessary — autograd in PyTorch/JAX/TensorFlow handles it correctly for any computation graph.

Use .grad to inspect gradients after loss.backward(). Use torch.autograd.gradcheck to verify custom operations. Use gradient clipping (torch.nn.utils.clip_grad_norm_) if gradients explode.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Backpropagation — Tutorial.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path