As convolutional neural networks grow deeper, an unexpected degradation problem emerges: once the network depth increases beyond a certain point, accuracy saturates and then rapidly degrades. Crucially, this degradation is not caused by overfitting. In an overfitted network, training error remains low while test error rises; in the degradation problem, adding more layers to a sufficiently deep model leads to higher training error as well as higher test error.

This behavior is distinct from the problem of vanishing or exploding gradients. While vanishing or exploding gradients historically prevented deep models from converging from the start of training, that issue has largely been addressed through normalized initialization methods and intermediate batch normalization layers, enabling networks with tens of layers to start converging with stochastic gradient descent.

Empirical tests on benchmark datasets demonstrate that degradation consistently occurs in standard "plain" networks (networks that simply stack feedforward layers). For instance, on CIFAR-10, a 56-layer plain network exhibits noticeably higher training and test errors than a 20-layer plain network. Similarly, on ImageNet, an 18-layer plain network achieves lower training and validation errors than its 34-layer plain counterpart throughout the entire training procedure.

The Network Degradation Phenomenon

Learn Before

Related

Learn After