Google

Language modeling is one of the application of feedforward networks: predicting upcoming words from prior word context. Neural language modeling
is an important NLP task in itself, and it plays a role in many important algorithms for tasks like machine translation, summarization, speech recognition, grammar correction, and dialogue.



Feedforward Neural Language Modeling

Neural language models have many advantages over the n-gram language models. Compared to n-gram models, neural language models can handle
much longer histories, can generalize better over contexts of similar words, and are more accurate at word-prediction. 

On the other hand, neural net language models are much more complex, slower to train, and less interpretable than n-gram models, so for many (especially smaller) tasks an n-gram language model is still the right tool.

Advantages vs. Disadvantages

Here’s the algorithm in detail for our mini example:
1. Select three embeddings from E: Given the three previous words, we look
up their indices, create 3 one-hot vectors, and then multiply each by the embedding matrix E. Consider wt−3. The one-hot vector for ‘for’ (index 35) is multiplied by the embedding matrix E, to give the first part of the first hidden
layer, the embedding layer. Since each column of the input matrix E is an 
embedding for a word, and the input is a one-hot column vector xi for word
Vi, the embedding layer for input w will be Exi = ei, the embedding for word
i. We now concatenate the three embeddings for the three context words to
produce the embedding layer e.

2. Multiply by W: We multiply by W (and add b) and pass through the ReLU
(or other) activation function to get the hidden layer h.

3. Multiply by U: h is now multiplied by U

4. Apply softmax: After the softmax, each node i in the output layer estimates
the probability P(wt = i|wt−1,wt−2,wt−3)

In summary, the equations for a neural language model with a window size of 3,given one-hot input vectors for each input context word, are:
e = [Ext−3;Ext−2;Ext−1]
h = σ(We+b)
z = Uh
yˆ = softmax(z) (7.22)
Note that we formed the embedding layer e by concatenating the 3 embeddings for the three context vectors; we’ll often use semicolons to mean concatenation of vectors.

Mini example

A widely cited and foundational study in neural language modeling was conducted by Bengio et al. in 2003. This work introduced a model that used a feed-forward neural network to estimate n-gram probabilities. The network was trained in an end-to-end fashion, and a key by-product of this process was the creation of distributed representations for words, which came to be known as word embeddings. This model was instrumental in demonstrating how neural networks could overcome the limitations of traditional statistical language models.

Bengio et al. (2003) Feed-Forward Neural Language Model

A language model is designed using a feedforward network architecture. It is trained to predict the next word by looking at a fixed-size window of the N preceding words (e.g., N=4). What is the most significant architectural limitation of this approach for modeling language?

Consider a feedforward neural network designed to predict the next word based on a fixed window of the three preceding words. Arrange the following computational steps in the correct order, from initial input to final output.

Based on the process described, what is the primary purpose of converting each input word into a 100-dimensional vector? Explain how this representation helps the model generalize its predictions to contexts it has not seen during training.

Learn Before

Related