A language model is generating the sequence of tokens 'the', 'dog', 'barked'. Write the mathematical expression for the joint probability of this entire sequence, `Pr(the, dog, barked)`, by decomposing it into a product of conditional probabilities.

Google

The joint probability of generating a specific sequence of tokens, such as `⟨s⟩ a b c d`, is calculated by applying the chain rule. This method decomposes the joint probability into a product of conditional probabilities, where each token's probability is conditioned on all the tokens that precede it. The expanded formula for this example is: $$Pr(⟨s⟩) \cdot Pr(a|⟨s⟩) \cdot Pr(b|⟨s⟩ a) \cdot Pr(c|⟨s⟩ a b) \cdot Pr(d|⟨s⟩ a b c)$$

Joint Probability of a Generated Sequence using the Chain Rule

A language model is generating a sequence of words. Given the following conditional probabilities, what is the joint probability of the model generating the exact sequence 'The cat sat'?

- The probability of starting with 'The' is 0.4.
- The probability of 'cat' following 'The' is 0.5.
- The probability of 'sat' following 'The cat' is 0.9.

Applying the Chain Rule to a Sequence

A language model has generated the prefix 'The weather is'. It is now deciding between two possible two-word completions. Based on the conditional probabilities provided in the case study, which completion is the model more likely to choose? Justify your answer by showing the calculation for each option.

Learn Before

Related