Interpreting Model Output Probabilities

An engineer is working with two different text-processing systems. System A generates a story one word at a time. To choose the word at position i, it calculates a probability distribution over the vocabulary based only on the words from position 1 to i-1. System B is used for a fill-in-the-blank task. Given a sentence with a missing word at position i, it calculates a probability distribution for that position using all other words in the sentence (both before and after position i)

Consider a model tasked with predicting a masked word within a complete sentence by looking at all surrounding words. The probability distribution calculated for this masked position has the same fundamental interpretation as the distribution from a model that generates a sentence one word at a time, where each new word is predicted based only on the words that came before it.