1Cademy - Acceptance-Rejection Mechanism for Speculative Decoding

Learn Before

Function of a Predicted Future Value Notation ( $q(\hat{y}_{i+t})$ )
Probability of a Predicted Future Value Notation ( $p(\hat{y}_{i+t})$ )

Activity (Process)

Acceptance-Rejection Mechanism for Speculative Decoding

In speculative decoding, a speculated token is accepted or rejected by comparing its probability from a draft model ( $q$ ) with that from a target model ( $p$ ). If the draft model's probability for a token, $q(\hat{y}_{i+t})$ , is greater than the target model's probability, $p(\hat{y}_{i+t})$ , the token is rejected with a probability of $1 - \frac{p(\hat{y}{i+t})}{q(\hat{y}{i+t})} $. In the alternative case, where$ q(\hat{y}{i+t}) \le p(\hat{y}{i+t})$, the token is accepted outright.