ModelRefs / Activation Functions (ReLU, Sigmoid, Softmax) — Tutorial
Activation Functions (ReLU, Sigmoid, Softmax) — Tutorial
The non-linearities that give neural networks their expressive power. Covers Why non-linearity is essential, Hidden layer activations — ReLU and its variants.
Overview
The non-linearities that give neural networks their expressive power
Level: Intermediate. Estimated reading time: 20 minutes.
Why non-linearity is essential
Without activation functions, a stack of linear layers collapses into a single linear transformation — no matter how many layers you have. Activation functions introduce non-linearity, allowing the network to learn curved decision boundaries, hierarchical features, and complex mappings.
Every hidden layer applies: output = activation(Wx + b). The activation's shape determines what patterns that layer can represent.
Hidden layer activations — ReLU and its variants
ReLU (Rectified Linear Unit): f(x) = max(0, x). Near-universal default for hidden layers. Simple, fast, sparse activations (negatives become zero). Problem: dying ReLU — if a neuron's input is always negative, its gradient is always zero and it stops learning.
Leaky ReLU: f(x) = x if x>0, else 0.01x. Fixes dying ReLU by allowing small negative gradient.
GELU (Gaussian Error Linear Unit): used in BERT, GPT. Smoother than ReLU; often slightly better for transformers.
Swish / SiLU: f(x) = x · σ(x). Used in modern CNNs and LLMs. Self-gated; smooth everywhere.
Output layer activations — match your task
Sigmoid: f(x) = 1/(1+e⁻ˣ) → output in (0,1). Use for binary classification output. Saturates at extremes — avoid in hidden layers (gradient vanishing).
Softmax: converts K raw logits to a probability distribution summing to 1. Use as the final activation for multi-class classification.
No activation (linear): use for regression outputs — you want unbounded real values.
Rule: match the output activation to your loss function. Sigmoid + binary cross-entropy. Softmax + categorical cross-entropy. Linear + MSE.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Activation Functions (ReLU, Sigmoid, Softmax) — Tutorial.