In a single step of a text generation process, a small, fast model proposes the candidate token sequence ['on', 'the', 'mat'] to extend the existing text ['The', 'cat', 'sat']. A larger, more accurate model then evaluates these candidates. The larger model accepts 'on' and 'the', but rejects 'mat'. After this rejection, the larger model's own prediction for the next token is 'rug'. What is the complete sequence of new tokens added to the text in this step?
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
In a single step of a text generation process, a small, fast model proposes the candidate token sequence
['on', 'the', 'mat']to extend the existing text['The', 'cat', 'sat']. A larger, more accurate model then evaluates these candidates. The larger model accepts'on'and'the', but rejects'mat'. After this rejection, the larger model's own prediction for the next token is'rug'. What is the complete sequence of new tokens added to the text in this step?Correcting a Step in a Generation Process
In a speculative decoding process, a draft model proposes a 3-token sequence, and the main evaluation model accepts all three tokens. Arrange the following actions in the correct chronological order to describe how the very next token is generated.