Learn Before
RNNLMScratch Class
The RNNLMScratch class implements an RNN-based language model from scratch by composing a previously defined RNN module with an output projection layer. It extends a Classifier base class and accepts an RNN instance, the vocabulary size, and a learning rate as constructor arguments. Because a language model's inputs and outputs are drawn from the same vocabulary, both share the same dimensionality, which equals the vocabulary size. The output layer is defined by a learnable weight matrix (where is the number of hidden units and is the vocabulary size), initialized from a scaled normal distribution, and a bias vector , initialized to zeros. These parameters project each hidden state to a vector of logits over the vocabulary.
0
1
Tags
D2L
Dive into Deep Learning @ D2L