Correcting a Step in a Generation Process
In a text generation process, a fast draft model proposes the token sequence ['over', 'the', 'lazy']. A more accurate evaluation model accepts the first two tokens, 'over' and 'the', but rejects the third token, 'lazy'. To continue generating the text, which model should be used to determine the very next token, and based on what preceding text sequence?
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
In a single step of a text generation process, a small, fast model proposes the candidate token sequence
['on', 'the', 'mat']to extend the existing text['The', 'cat', 'sat']. A larger, more accurate model then evaluates these candidates. The larger model accepts'on'and'the', but rejects'mat'. After this rejection, the larger model's own prediction for the next token is'rug'. What is the complete sequence of new tokens added to the text in this step?Correcting a Step in a Generation Process
In a speculative decoding process, a draft model proposes a 3-token sequence, and the main evaluation model accepts all three tokens. Arrange the following actions in the correct chronological order to describe how the very next token is generated.