A language model is processing the input sequence 'The cat sat on the'. The model's vocabulary consists of 10,000 unique words (tokens). Describe the direct output that the model's final layer produces to determine the next word. Specifically, what is the structure and size of this output, and how does it relate to the model's vocabulary?

Google

Deep neural networks, such as a parameterized Transformer decoder denoted as $$\mathrm{Decoder}_{\theta}(\cdot)$$, generate a probability distribution for the next token based on a sequence of preceding tokens, $$x_0, \dots, x_i$$. This predicted distribution is represented as $$\mathrm{Pr}_{\theta}(\cdot|x_0, \dots, x_i)$$, which is often abbreviated as $$\mathbf{p}_{i+1}^{\theta}$$. The model's final output for that position is typically the token that receives the maximum probability.

Neural Network-Based Next-Token Probability Distribution

A neural network language model, which has a vocabulary of 50,000 unique tokens, is given the input context 'The sun is shining and the sky is'. What does the model's final layer compute and output directly to represent the likelihood of the next token?

Language Model Output Structure

Based on the provided scenario, explain what the vector of 10,000 numbers represents and what the specific value of 0.65 at the index for the token 'mat' signifies.

Learn Before

Related