The Role of Randomness in Token Acceptance
In a system that uses a faster draft model to propose tokens for a more accurate target model, a random number is drawn from a uniform distribution between 0 and 1 to help decide whether to accept a proposed token. Explain the fundamental reason for using this random number instead of a fixed threshold. What would be the consequence of always using a fixed value, such as 0.5, for this decision?
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Formula for the Number of Consecutively Accepted Tokens in Speculative Decoding
In a system that uses a faster, smaller model to generate candidate tokens for a larger, more accurate model, a single token is being evaluated. The faster model assigns a probability of 0.8 to this token, while the more accurate model assigns it a probability of 0.6. For the acceptance check, a random number of 0.7 is drawn from a uniform distribution between 0 and 1. Based on this information, what is the outcome for this candidate token?
Speculative Decoding Acceptance Analysis
The Role of Randomness in Token Acceptance