Example of Draft Token Generation in Speculative Decoding
To illustrate the draft generation phase, which is the initial step in speculative decoding, consider a scenario where the draft model predicts a sequence of candidate tokens. Starting with a given prefix , the draft model utilizes its probability distribution to autoregressively generate a sequence of five tokens: . Each token in this draft sequence is predicted based on the initial prefix and all previously generated draft tokens within the current step.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Example of Draft Token Generation in Speculative Decoding
A system uses a fast draft model to autoregressively generate a sequence of several candidate tokens from a given prefix. The model generates these candidates one by one, and for each step, it greedily selects the token with the highest probability according to its own distribution,
Pr_q. If the system is in the process of generating the third candidate token in the sequence,ŷ_{i+3}, which of the following represents the correct set of information the draft model's probability distribution must be conditioned on for this specific step?A developer is implementing the draft token generation phase of a text generation system. The system is designed to autoregressively produce a short sequence of candidate tokens at each step. The developer's code for generating the third token in a sequence,
ŷ_{i+3}, incorrectly conditions the draft model's probability distribution only on the initial prefix[X, y_{≤i}]and the first candidate tokenŷ_{i+1}, omitting the second candidate tokenŷ_{i+2}from the context. What is the most likely consequence of this specific error?A fast, approximate language model is tasked with generating a sequence of three candidate tokens (ŷᵢ₊₁, ŷᵢ₊₂, ŷᵢ₊₃) starting from a given text prefix P. Arrange the following actions in the correct chronological order to describe how this sequence is produced.
Learn After
A small, fast language model is used to generate a sequence of 3 candidate words. Starting with the initial context 'The cat sat on the', arrange the following actions in the correct chronological order to produce the three-word draft sequence.
A small, fast language model is used to generate a sequence of 3 candidate tokens. It generates one token at a time, and each new token prediction is based on the original context plus any tokens it has already generated in the current sequence. Given the model's predictions below, what is the final 3-token sequence it will generate starting from the context 'The ocean is'?
- When given 'The ocean is', the model's top prediction is 'blue'.
- When given 'The ocean is blue', the model's top prediction is 'and'.
- When given 'The ocean is blue and', the model's top prediction is 'deep'.
- When given 'The ocean is deep', the model's top prediction is 'water'.
Debugging Draft Token Generation