Based on the principles of auto-regressive generation, explain this specific error. In your explanation, describe how the probability of generating 'bancaire' is calculated and why the model might have selected it over the correct word, despite the original input sentence.

Google

In sequence-to-sequence models, the probability of generating a specific output token is conditioned on both the entire input sequence and all previously generated output tokens. This is represented by the formula $Pr(y_i | x_1, ..., x_m, y_1, ..., y_{i-1})$, where $y_i$ is the current output token, $(x_1, ..., x_m)$ is the complete input sequence, and $(y_1, ..., y_{i-1})$ are the output tokens already generated. This conditional probability is the core calculation performed at each step of the auto-regressive generation process.

Conditional Probability in Sequence-to-Sequence Generation

A language model generates an output sequence one token at a time, where each new token's probability depends on prior information. If the model has already produced the first three tokens of an output based on a given input sequence, which of the following best describes the complete set of information used to calculate the probability for the *fourth* token?

A team is developing two different text generation models, Model A and Model B. Both are given the same input phrase, "The sky is".

- **Model A** calculates the probability for the next word (e.g., "blue") based on the full input phrase "The sky is".
- **Model B** calculates the probability for the next word (e.g., "blue") based on the full input phrase "The sky is" AND any words it has already generated in the current output sequence.

One of these models represents a fundamentally flawed approach for generating coherent, multi-word sequences. Identify which model is flawed and explain why its method for determining the probability of the next word is problematic for generating longer, sensible sentences.

Learn Before

Related