In a residual network, the desired underlying mapping $$f(\mathbf{x})$$—the function the network ultimately aims to approximate—is not learned directly by a stack of layers. Instead, those layers are reformulated to learn only the residual mapping $$g(\mathbf{x}) = f(\mathbf{x}) - \mathbf{x}$$, and the target function is recovered as $$f(\mathbf{x}) = g(\mathbf{x}) + \mathbf{x}$$. This reformulation is motivated by the degradation problem: as plain networks grow deeper, their training accuracy can paradoxically worsen, suggesting that the added layers struggle to approximate even the identity function. By recasting the problem in terms of $$g(\mathbf{x})$$, the identity case $$f(\mathbf{x}) = \mathbf{x}$$ reduces to $$g(\mathbf{x}) = 0$$, which is significantly easier for a network to learn because it only requires driving the weights and biases of the constituent layers toward zero.

Residual Mapping

The degradation of training accuracy reveals an optimization paradox when comparing a shallower architecture to a deeper counterpart that adds extra layers. Because the solution space of the shallower model is a subspace of the deeper model, the deeper architecture should theoretically never yield higher training error than the shallower one.

This principle can be shown by construction: consider a deeper model where the initial layers are directly copied from a learned shallower model, and all additional layers are configured as identity mappings. By construction, this configuration produces the exact same function and training error as the learned shallower model.

Despite the existence of this feasible solution, standard gradient-based optimization solvers fail to locate it. In practice, optimizers applied to plain networks cannot find solutions that are comparably good or better than the identity-constructed solution within feasible training time, showing that plain deep architectures are fundamentally difficult to optimize.

The Identity Mapping Construction Paradox

Learn Before

Related

Learn After