Optimization Difficulties in Deep Plain Networks
Investigations into deep plain networks show that the optimization failure underlying degradation cannot be attributed to vanishing signals. Plain networks trained with batch normalization maintain forward propagated signals with non-zero variances and backward propagated gradients with healthy norms, meaning signals neither vanish nor explode during backpropagation.
Furthermore, the degradation problem cannot be resolved simply by extending training duration. Experiments running up to three times () more training iterations still exhibit persistent degradation, suggesting that deep plain networks suffer from exponentially low convergence rates that hinder reduction of training error.
These optimization difficulties suggest that standard solvers struggle to approximate identity mappings through stacks of nonlinear layers. If an identity mapping is optimal or near-optimal, driving stacked nonlinear layers to act as identity functions proves significantly harder for current optimizers than learning perturbations referenced to an explicit identity shortcut.
0
1
Tags
Prep Sessions
Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Ch.3 Deep Residual Network Architecture - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
The Degradation Problem in Deep Networks - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Learn After
What empirical observation demonstrates that the degradation problem in deep plain networks trained with batch normalization is not caused by vanishing or exploding signals?
Tripling the number of training iterations () effectively resolves the degradation problem in deep plain networks.
What specific form of convergence rate impedes training error reduction in deep plain networks suffering from the degradation problem?
Contrast the challenge standard optimizers face when an identity mapping is optimal in deep plain networks versus when using an explicit identity shortcut.