Analyzing a Sequence Construction Method
An engineer is implementing a text generation algorithm. They describe their process for extending a sequence as follows: 'At step i, my current sequence is y_1...y_{i-1}. To find the next token, y_i, I identify the token that maximizes the conditional probability Pr(y_i | y_1...y_{i-1}). However, before I finalize the new sequence y_1...y_i, I compare its overall score to the score of a different, shorter sequence that I discarded at step i-2 to ensure I am still on the absolute best path overall.'
Analyze the engineer's process. Does the final step of their method (comparing the current sequence's score to a previously discarded path) correctly represent how the single optimal sequence is constructed in a greedy search? Explain your reasoning.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Formula for the Candidate Set in Greedy Search
A language model is generating text one token at a time by always selecting the single most probable next token. It has already produced the sequence 'The sun is shining'. For the very next step, the model calculates the following conditional probabilities for the next token:
- P(brightly | 'The sun is shining') = 0.55
- P(today | 'The sun is shining') = 0.25
- P(and | 'The sun is shining') = 0.15
- P(down | 'The sun is shining') = 0.05
Based on this method of construction, what will the updated sequence be after this step?
A language model generates text by always appending the single most probable token given the sequence generated so far. Arrange the following steps to correctly illustrate how the model would construct the three-token sequence 'The quick fox'.
Analyzing a Sequence Construction Method