Learn Before
Evaluating Draft Model Effectiveness
A research team is using a large, powerful language model to generate text. To speed up this process, they use a smaller 'draft' model to propose a short sequence of words, which the large model then verifies. If the large model would have generated the same sequence, it accepts the draft's proposal, saving significant time. If not, it rejects the proposal and generates a single word itself, which is a slow fallback operation. The team is testing two draft models. Based on the descriptions below, which draft model is likely to result in a greater overall increase in text generation speed for the combined system? Justify your choice.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Evaluation in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A team is building a system to accelerate text generation from a very large, high-quality, but slow language model. Their strategy involves using a much smaller, faster 'draft' model to propose a sequence of words first. The large model then reviews this draft sequence; if the sequence is plausible, the large model accepts it, saving time. If not, the large model rejects it and generates its own sequence from scratch. To maximize the overall speed of the system (words generated per second), which property is most desirable for the draft model's probability distribution over the next words?
Evaluating Draft Model Effectiveness
Optimizing a Two-Model Generation System