Probability Normalization over a Candidate Set

An autoregressive model is given an input prompt, x, which is the sequence 'The best movie I ever saw was'. The model has already generated the partial output sequence, y_{<i}, which is 'about a'. The model's next task is to predict the probability of the next token, y_i, based on the standard conditional probability notation Pr(y_i|x, y_{<i}). What is the actual, full sequence of tokens the model uses as its context to make this prediction?

In the context of autoregressive sequence generation, the notation Pr(y_i|x, y_{<i}) implies that the model treats the input x and the previously generated tokens y_{<i} as two separate, distinct sources of information for predicting the next token y_i.

Interpreting Autoregressive Model Inputs