Mathematical Formulation of Verification Model Evaluation in Speculative Decoding

In a system designed to accelerate text generation, a smaller 'draft' model proposes a sequence of tokens, which are then checked by a larger 'verification' model. Consider the following state:

• The initial input text is: The solar system has
• The sequence of already verified and accepted tokens is: eight planets. The largest is
• The draft model now proposes the next three tokens as: Jupiter, ,, a

To evaluate the third proposed token (a), what is the complete set of information the verification model conditions its probability calculation on?

In a text generation process using a draft model and a verification model, the system is at step i. The draft model proposes a sequence of new tokens: ŷ_{i+1}, ŷ_{i+2}, ŷ_{i+3}. The verification model, p, must now calculate the probability for each of these draft tokens. Which of the following mathematical expressions correctly represents the information the verification model conditions on to calculate the probability of the third draft token, ŷ_{i+3}? (Let X be the original input and Y_{≤i} be the sequence of already verified tokens.)

Analyzing a Flawed Verification Process in Text Generation