ModelRefs / Calculus & Derivatives — Tutorial
Calculus & Derivatives — Tutorial
The chain rule and partial derivatives that make neural network training possible. Covers Derivatives: the rate of change.
Overview
The chain rule and partial derivatives that make neural network training possible
Level: Intermediate. Estimated reading time: 25 minutes.
Derivatives: the rate of change
A derivative measures how much a function's output changes when you nudge its input. For f(x) = x², the derivative f'(x) = 2x tells you: at x=3, a tiny increase in x increases f by about 6× that amount.
In machine learning, the derivative answers: "If I increase this weight by a tiny amount, how much does the loss increase or decrease?" A negative derivative means increasing the weight reduces the loss — so you should increase it. A positive derivative means increasing the weight raises the loss — so you should decrease it. This is exactly gradient descent.
The derivative of common functions: - f(x) = xⁿ → f'(x) = nxⁿ⁻¹ - f(x) = eˣ → f'(x) = eˣ (e is its own derivative — why it appears everywhere) - f(x) = ln(x) → f'(x) = 1/x
The chain rule: how backpropagation works
A neural network is a composition of functions: output = f₄(f₃(f₂(f₁(x)))). To train it, you need the derivative of the loss with respect to each weight, deep inside the composition.
The chain rule says: d/dx[f(g(x))] = f'(g(x)) · g'(x). In English: the derivative of a composition is the product of the derivatives at each step.
For a 3-layer network with loss L, layer outputs h₁, h₂, h₃: dL/dW₁ = (dL/dh₃) × (dh₃/dh₂) × (dh₂/dh₁) × (dh₁/dW₁)
Backpropagation is just the chain rule applied systematically from the output layer back to the input. Each layer computes its local gradient and passes it backward. PyTorch's autograd does this automatically.
Partial derivatives and the gradient vector
Most functions in ML have many inputs — a model with 7 billion parameters has 7 billion inputs to the loss function. The gradient is the vector of all partial derivatives: ∇L = [∂L/∂w₁, ∂L/∂w₂, ..., ∂L/∂wₙ].
A partial derivative ∂L/∂wᵢ measures how the loss changes when only wᵢ is varied, holding everything else constant. Computing all partial derivatives at once is what one backward pass does.
The gradient points in the direction of steepest ascent. Gradient descent steps in the opposite direction: w ← w - α·∇L, where α is the learning rate. This moves the weights in the direction that reduces the loss.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Calculus & Derivatives — Tutorial.