Trade-offs of Greedy Search in Inference
Greedy search provides an efficient and easily implementable strategy for large language model inference, making it highly attractive for tasks where speed and simplicity are paramount. However, its primary drawback is its suboptimal nature; because it makes locally optimal choices, it risks pruning potentially high-quality sequences during the early stages of decoding. For applications requiring superior search results, strategies that explore multiple paths simultaneously, such as beam search, are often preferred.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Formula for a Candidate Solution Set
A developer is building a text generation system to find the single most probable 20-word summary of a given document. Their proposed method is to generate every possible 20-word sequence, calculate the probability of each one, and then select the sequence with the highest probability. Which statement best analyzes the fundamental limitation of this approach?
Selecting a Generation Strategy for a Real-Time Application
The Role of Search Algorithms in Text Generation
An engineer is developing a system to generate a 10-word sentence. They are comparing two approaches:
- Approach 1: Systematically generate every possible 10-word sequence from the system's vocabulary, calculate a probability score for each, and select the one with the highest score.
- Approach 2: Generate the first word, then, given that first word, generate the most probable second word, and so on, making a locally optimal choice at each of the 10 steps.
Which statement best analyzes the relationship between these two approaches?
Trade-offs of Greedy Search in Inference
Learn After
Evaluating an Inference Strategy for a Creative Writing Assistant
A language model is generating a response, starting with the phrase 'The best way to learn is...'. The model must decide on the next three words. At each step, it uses a strategy where it always chooses the single word with the highest probability. Given the probabilities below, which sequence of three words will the model generate?
Step 1 (after '...is'):
by: 0.5through: 0.3with: 0.2
Step 2 (if 'by' is chosen):
doing: 0.6reading: 0.3listening: 0.1
Step 3 (if 'by doing' is chosen):
it: 0.4the: 0.5a: 0.1
Analyzing a Step-by-Step Text Generation Strategy