Calculating Conditional Probability in a Generation Step
An auto-regressive language model is generating a sequence. After two steps, it has produced the sequence <s> The cat with a cumulative probability of 0.15. In the third step, it adds the token sat, and the new cumulative probability for the sequence <s> The cat sat becomes 0.09. What was the conditional probability the model assigned to the token sat at this step, i.e., what is Pr(sat | <s> The cat)? Show your calculation.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
An auto-regressive language model using a greedy decision rule has generated the sequence 'The cat sat' with a cumulative probability of 0.2. At the next step, the model calculates the following conditional probabilities for the next token: P('on' | 'The cat sat') = 0.6, P('by' | 'The cat sat') = 0.3, and P('under' | 'The cat sat') = 0.1. What will be the newly generated sequence and its updated cumulative probability?
A language model is tasked with generating the three-token sequence 'The quick brown' using a greedy, auto-regressive approach. Arrange the following actions in the correct chronological order that the model would take.
Calculating Conditional Probability in a Generation Step