The degradation of training accuracy reveals an optimization paradox when comparing a shallower architecture to a deeper counterpart that adds extra layers. Because the solution space of the shallower model is a subspace of the deeper model, the deeper architecture should theoretically never yield higher training error than the shallower one.

This principle can be shown by construction: consider a deeper model where the initial layers are directly copied from a learned shallower model, and all additional layers are configured as identity mappings. By construction, this configuration produces the exact same function and training error as the learned shallower model.

Despite the existence of this feasible solution, standard gradient-based optimization solvers fail to locate it. In practice, optimizers applied to plain networks cannot find solutions that are comparably good or better than the identity-constructed solution within feasible training time, showing that plain deep architectures are fundamentally difficult to optimize.