This diagram provides a visual representation of Permuted Language Modeling, where a sequence is generated in a non-sequential, permuted order. In this example, the prediction order is $x_0 \rightarrow x_4 \rightarrow x_2 \rightarrow x_1 \rightarrow x_3$. Each row illustrates a step in the generation process, where the blue squares indicate the tokens that have already been generated and are used as context for predicting the next token. The step-by-step conditional probabilities are shown on the right:

1. Step 1 (Predict $x_0$): The process starts with $x_0$, which is treated as a given starting point. Its probability is set to 1: $\text{Pr}(x_0) = 1$.
2. Step 2 (Predict $x_4$): The model predicts $x_4$ conditioned on the embedding of $x_0$: $\text{Pr}(x_4|\mathbf{e}_0)$.
3. Step 3 (Predict $x_2$): The model predicts $x_2$ conditioned on the embeddings of the already generated tokens, $x_0$ and $x_4$: $\text{Pr}(x_2|\mathbf{e}_0, \mathbf{e}_4)$.
4. Step 4 (Predict $x_1$): The model predicts $x_1$ using the context of $x_0, x_4,$ and $x_2$: $\text{Pr}(x_1|\mathbf{e}_0, \mathbf{e}_4, \mathbf{e}_2)$.
5. Step 5 (Predict $x_3$): Finally, the model predicts $x_3$ conditioned on all other tokens: $\text{Pr}(x_3|\mathbf{e}_0, \mathbf{e}_4, \mathbf{e}_2, \mathbf{e}_1)$.