ModelRefs / Residual Connection — AI Glossary

Residual Connection — AI Glossary

A skip connection that adds a layer's input directly to its output, enabling deep networks to train without vanishing gradients.

Overview

Introduced in ResNet (He et al. 2015) and adopted universally in transformers. Each transformer sub-layer (attention, feed-forward) wraps its computation with a residual: output = LayerNorm(x + sublayer(x)). Residuals make gradient flow stable at depth >100 layers and are fundamental to the transformer's scalability.

Reference details

Topic	architecture
Also known as	skip connection, residual stream
Last reviewed	2026-06-24

Primary source

He et al., Deep Residual Learning for Image Recognition (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Residual Connection — AI Glossary.

Frequently asked questions

What is Residual Connection?

A skip connection that adds a layer's input directly to its output, enabling deep networks to train without vanishing gradients.

Is Residual Connection the same as skip connection?

Yes — skip connection, residual stream are common aliases for Residual Connection.

What concepts are related to Residual Connection?

Closely related concepts include layer normalization, transformer, feed forward network.