Post-Acceptance Token Generation in Speculative Decoding

Set of Accepted Draft Tokens

Set of Tokens Generated in a Single Speculative Decoding Step

In a text generation process designed for speed, an initial sequence ['The', 'cat', 'sat'] is extended. A fast proposal mechanism suggests the candidate tokens ['on', 'the', 'mat']. A more accurate, final-check mechanism then processes these candidates and produces the final, complete sequence: ['The', 'cat', 'sat', 'on', 'the', 'rug']. Based on this outcome, how many of the candidate tokens were accepted before the final-check mechanism generated its own token?

In a text generation process that uses a fast model to propose candidate tokens and a more accurate main model to check them, a single generation step has just completed. Arrange the following components to correctly represent the structure of the full, updated text sequence.

Visual Representation of a Speculative Decoding Step's Output

Analyzing a Speculative Generation Step