Google

In language modeling, the ground-truth distribution at a given position, denoted as $$\mathbf{p}_{i+1}^{\mathrm{gold}}$$, is defined as the one-hot representation of the actual next token, $$x_{i+1}$$. This one-hot vector acts as the exact target for the model's prediction at that step.

Ground-Truth Distribution as a One-Hot Representation

In the context of training a language model, representing the ground-truth distribution as a one-hot vector implies that the training process considers all incorrect tokens to be equally wrong, regardless of their semantic similarity to the correct token.

In the training process for a language model, the ground-truth distribution for the correct next token is represented as a vector containing a single '1' and zeros for all other entries. In your own words, explain what the position of the '1' signifies and what this vector structure implies about the training target.

Explaining the Ground-Truth Vector

A language model is being trained on a text corpus where it learns to predict the next word in a sequence. The model's entire vocabulary is ordered as follows: `['a', 'bright', 'day', 'is', 'shining']`. If the model is given the input context 'a bright' and the actual next word in the training data is 'day', which vector correctly represents the ground-truth target for this specific training step?

Learn Before

Related