Acceptance and Rejection Criteria for Speculated Tokens
In speculative decoding, the decision to accept or reject a speculated token depends on the probabilities assigned by the draft model, , and the verification model, . If , the speculation is accepted. By contrast, if , the speculation is rejected with a probability of . This mechanism determines the maximum number of consecutively accepted tokens.
0
1
Tags
Foundations of Large Language Models
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Formula for the Number of Consecutively Accepted Tokens in Speculative Decoding
Post-Acceptance Token Generation in Speculative Decoding
In an accelerated text generation method, a sequence of candidate tokens is proposed and then individually verified. The verification results for a sequence of 5 tokens, in order, are: [Accepted, Accepted, Rejected, Accepted, Accepted]. According to the rules of this method, a continuous block of accepted tokens from the beginning of the sequence is appended to the final output, and the process halts at the first rejected token. How many tokens from this proposed sequence will be appended to the final output?
Evaluating a Speculative Decoding Step
Diagram of Post-Acceptance Token Prediction in Speculative Decoding
Rationale for Consecutive Acceptance in an Accelerated Generation Method
You are implementing speculative decoding in a cus...
In a production LLM service using speculative deco...
You are reviewing logs from a production LLM endpo...
Diagnosing a Speculative Decoding Slowdown in Production
Choosing τ and Model Roles for Low-Latency Speculative Decoding
Tuning Speculative Decoding Under a Fixed Verification Budget
Designing a Speculative Decoding Control Policy for a Latency-Sensitive Product
Root-Causing Low Speedup Despite Parallel Verification
Explaining a “Fast but Wrong” Speculative Decoding Regression
Interpreting a Speculative Decoding Trace and Identifying the Bottleneck
Acceptance and Rejection Criteria for Speculated Tokens