Learn Before
Resource Consumption in Text Generation
A developer is tuning a text generation system. They have two configurations:
- Configuration A: At each step of generating text, the system considers only the 3 most likely next words.
- Configuration B: At each step, the system considers the 100 most likely next words.
Describe two specific computational resources that will be more heavily consumed by Configuration B compared to Configuration A, and briefly explain why for each.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Diminishing Returns in Search Scaling
Optimizing Inference Performance
An engineer modifies a language model's inference procedure to evaluate a significantly larger number of potential output sequences at each generation step, aiming to enhance the final output quality. What is the most direct and unavoidable trade-off associated with this modification?
Resource Consumption in Text Generation