Learn Before
Dropout Placement in Deep RNNs
In a deep (multilayer) Recurrent Neural Network (RNN), dropout regularization is applied to the intermediate hidden states passed between the stacked layers. Specifically, a dropout operation is introduced after the output of each recurrent layer before it serves as the input to the subsequent layer. However, dropout is conventionally omitted after the final recurrent layer. While high-level deep learning APIs abstract this structural logic through a simple dropout parameter that automatically places dropout between layers, implementing deep RNNs in minimalist frameworks requires developers to explicitly inject a dropout operation after every RNN layer except the last one during the iterative forward computation.
0
1
Tags
D2L
Dive into Deep Learning @ D2L