A Decade of Residuals: History & Effects on modern ML

A decade ago training deep neural nets was quite the bottleneck in ML. Increasing depth, counter-intuitively, reduced performance. although we knew that depth meant better representation in theory, but in practice training was brittle and numerically unstable as the chain of jacobians gets longer and longer, many work-arounds existed such as layer-wise pre-training, semi-supervised stacking, better initializations …

until a different approach succesfully mitigated a major design flaw in pure sequential neural nets: the vanishing gradient problem, espeically pronounced when sigmoid and tanh a common activation functions choice at that time.

Residual connections were introduced to solve that problem but over time this solution influenced a wide range of design choices, training strategies and representation learning dynamics.

note: figures in this blog reflect a focus on the residual stream(s) by deviating from usual illustrations, and ommittes normalization layers in most figures for simplicity

the high level intuition of residual connections is that, when training very deep networks (e.g MLPs), the change of one layer’s representation compared to the next would be less and less different and thus instead of fully learning the representation at layer $\mathrm{L+1}$ we can simply learn the difference / change relative to layer $\text{L's}$ representation.

Although the computed function class remains unchanged, during training the latter formulation offers a great optimization advantage.

if we take a closer look at a single param tensor in a certain layer in a deep MLP and derive it’s gradient computation we’ll end-up with something like:

The premise of gradient-based methods is that useful gradietn signal vanishes only at optima, however in practice, a single jacobian factor with a small norm is sufficient to collapse the entire product effectively blocking learning in earlier layer and preventing gradient flow

intuitively if a parameter can affect loss only through it’s effect on the next layer, then shrinking gradient downstream removes its only pathway to contribute to optimization.

A common cause of this phenomenon is the presence of saturation regions in activation functions (e.g extreme magnitude inputs to sigmoid & tanh, …)

residual blocks introduce an explicit identity term into the layer to layer jacobian ensuring a theoretically depth-independent gradient component

the first major breakthrough using this idea was ResNet, which achieved SOTA performance on ImageNet at the time and demonstrated, for the first time, that networks hundreds of layers deep could be trained reliably.

beyond training stability ResNet proved the mental model of information flow and representation evolution though model depth. it shifted the view from semantically disassociated blocks into a continuous flow of updates with each layer restircted by design to learn small deviations representation-wise and utilize depth to iteratively move representation toward the target underlying manifold.

Transformers later adopted the same principle modelizing the entire model as a sequence of residual updates, where attention modules and MLPs contribute to the shared residual stream. this opened up an entirely new design space beyond vanilla feed-forward stacks: how to combine residuals with normalization layers, how strongly to consider each update and how to keep the evolving representation well-conditioned across however-many blocks we used.

by introducing another stream of representation flow, normalization, selection, and filtering became necessary to keep representation updates stable and meaningful. the simplest form of such control are gates:

where $g$ regulate the residual connection update. this idea appeared in architectures such GRU LSTM in sequential modeling, similar approaches appeared in CNN and UNet variants (e.g nnUNet, …) where skip connection aren’t simply added to the main stream but rather gated or attention weighted allowing fine-grained control on information flow.

a more structured extension is mixture of experts (MoE) paradigm. instead of a single residual update, multiple specialized blocks (experts) propose updates and a learned router selects or weights them. in residual terms, MoEs decide which transformations are allowed to write into the shared stream for a given input.

another approach focuses on modulating residual updates rather than selecting them, feature-wise linear modulation (FiLM) applies a shifted and scaled input-conditioned transformation:

allowing more dynamic updates based on input features, structure or other condition vectors (very commonly used in diffusion and generative models)

even modern activation functions like SwiGLU and GEGLU could be interpreted as following the input-conditioned gate pattern, all cited mechanisms and components manifest how simple residuals turned into controlled editing process over a shared stream

a natural interpretation of a continuous residual update flow is that of a dynamical system. once models are written as

depth can be interpreted as time, a formulation studied for decades across physics, control and robotics

a standard ResNet corresponds to that with $\Delta t = 1$.

in such interpretation each residual block computes a small update based on the current state and moves the representation forward along a trajectory in feature space. instead of stacking transformations, the network is learns a discretized vector field.

this perspective has since evolved into families of models such as Neural ODEs, continuous-time RNNs and liquid neural networks, which learn an internal state flows over time. companies like liquid.ai build directly on this idea to achieve highly efficient models in several settings.

the Neural ODEs paper authors proceed to work adaptive time discretization beyond ResNet-like models using specialized solvers to integrate learned dynamics

once models are built around a residual stream a new regularization technique emerges: Drop Path

ensuring no subset of blocks dominate the updates contribution; during training we randomly skip layers / blocks following a probability $p_{drop\_path}$, further reinforcing the representation similarity block-to-block and forces later blocks to approximate earlier dynamics under stochastic depth

the authors set and proved their block recurrence hypothesis by training a vision transformer (distilled from Dinov2) using only 2 blocks, each representing a set of functionally similar blocks in the teacher model, applying each recurrently and preserved 96% of Dinov2’s linear-probe accuracy on ImageNet-1k.

dynamic-systems-like interpretability was also studied in the same work, finding tokens converging to angular attractors and local self-corrective behaviors tested by robustness to angular perturbations

while earlier augmentations focused on expanding residual depth, most notably the Deep Layer Aggregation paper (Yu et al) enfluencing many subsequent work, recent efforts shifted toward expanding residual width. in early 2025 a bytedance team introduced hyper connections (Zhu et al) as an expansion of vanilla residual, offering an learnable interpolation between pre-norm and post-norm residual connections as well as boarder mechanisms for width expansion methods (ResiDual, Parallel Transformer Blocks, …).

since the original transformers paper, many architectural tweaks has been adopted, most prominently post-norm variants. yet the latter didn’t offer a free-lunch; tho it mitigated vanishing gradients it resulted in more common representation collapse, whereas pre-norm exhibits the opposite trade-off.

the core idea behind hyper connections is the use multiple residual streams, a merging matrix $\mathrm{H}^{pre}$ to aggregate stream at the input of each layer, a redistribution matrix $\mathrm{H}^{post}$ redistribute the update at the layer output and a residual mixing matrix $\mathrm{H}^{res}$ mixes pre- and post- layer residual streams

the hyper connection framework therefore introduces both depth connections: pre- and post- layer aggregation and redistribution, and width connections: residual streams mixing.

in the original formulation of the $\mathrm{H}^{pre}$, $\mathrm{H}^{post}$ and $\mathrm{H}^{res}$ are defined as the sum of two matrices, a learnable static “bias” matrix and the an input-conditioned “dynamic” matrix (a scaled non-linear transformation of input):

tho carefull initialization of each matrix leads to stable training as reported in the original paper, at larger scale behavior got more subtle.

a recent paper from deepseek, mHC: manifold-constrained Hyper Connections (Xie et al) idetify a critical design flaw in the hyper connections framework: breaking the identity mapping property that underlies the stability of residual connections

looking at the unrolled equation of hyper connections of non-consecutive layers reveals an exponential effect of the repeated multiplication by $\mathcal{H}^{res}$ disturbing identity mapping, for instance positive entries $\lt 1$ lead to exponential decay, positive entries $\gt 1$ result in exponential growth and negative entries result in alternating oscillations, none of which are norm contolled.

to address that, the mHC paper proposes constraining the mixing matrix $\mathcal{H}^{res}$ to be positive, with each row and column summing to 1 to preserve the originally intended linear mixing behavior while maintaining stable identity transport and well conditioned gradients.

the proposed constriant places $\mathcal{H}^{res}$ in the Birkhoff polytope i.e set of matrices with positive elements, each row and column summing to 1, formally known as “doubly stochastic matrices”:

projection into the Birkhoff polytope is performed using the sinkhorn-knopp algorithm (iterative dim normalization) applied to the element-wise exponential of $\mathcal{H}^{res}$.

moreover this constrain admits a compositional closure i.e the product $\large\prod_{i=l}^{l+n}\mathcal{H}^{res}_l$ remains doubly stochastic, enabling norm-preserving residual transport across depth

following the same norm-preserving principle,the mHC paper additionally enforces positivity on $\mathcal{H}^{pre}$ and $\mathcal{H}^{post}$ by using sigmoid non-linearity

together resulting in the constrained formulation:

implementation note: the deepseek team reported only $\approx 6.7\%$ compute overhead enabled by selective activation checkpointing, specialized fused kernels and communication-compute overlap in their DualPipe setup and directly addressed the memory overhead introduced by the multiple residual streams unlike the original hyper connections paper.

Throughout the years, mental models in deep learning have changed in many diverse and interesting ways, yet in most of them residual connections persisted. As more top venue work highlights the similarities between evolving architectural models in machine learning and dynamical systems, we are gradually breaking previously established scaling laws through more sophisticated geometric structure and constrained representation dynamics

the following papers constitute the main references for the ideas discussed in this article