An engineer is optimizing a text generation system that uses a large, powerful model for final output. To speed up the process, they are testing two different smaller 'draft' models to propose sequences of tokens for the large model to verify.
- Draft Model X: Generates 5 candidate tokens in 10ms. On average, the large model accepts only 1 of these 5 tokens.
- Draft Model Y: Generates 5 candidate tokens in 20ms. On average, the large model accepts 4 of these 5 tokens.
Assuming the veri
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
An engineer is optimizing a text generation system that uses a large, powerful model for final output. To speed up the process, they are testing two different smaller 'draft' models to propose sequences of tokens for the large model to verify.
- Draft Model X: Generates 5 candidate tokens in 10ms. On average, the large model accepts only 1 of these 5 tokens.
- Draft Model Y: Generates 5 candidate tokens in 20ms. On average, the large model accepts 4 of these 5 tokens.
Assuming the veri
Optimizing Chatbot Latency
Draft Model Selection Rationale