Next Token Selection in an Accelerated Decoding Process

In an accelerated text generation process, a sequence has been extended. The confirmed prefix is 'The cat sat on the', and two subsequent tokens, 'mat and', have just been accepted. The system now needs to generate the very next token. The underlying evaluation model provides the following probabilities for potential next tokens, given the full context 'The cat sat on the mat and':

P('looked') = 0.55 P('slept') = 0.25 P('waited') = 0.15 P('the') = 0.05

According to the principle of selecting t

In a speculative decoding process, after a sequence of n draft tokens has been verified and accepted, the very next token (at position n+1) is generated by selecting the most likely token according to the draft model's probability distribution.