Google

Greedy search provides an efficient and easily implementable strategy for large language model inference, making it highly attractive for tasks where speed and simplicity are paramount. However, its primary drawback is its suboptimal nature; because it makes locally optimal choices, it risks pruning potentially high-quality sequences during the early stages of decoding. For applications requiring superior search results, strategies that explore multiple paths simultaneously, such as beam search, are often preferred.

Trade-offs of Greedy Search in Inference

Evaluate the suitability of the team's chosen text generation method for their stated goal. Justify your reasoning by describing the likely characteristics of the generated text and explaining why those characteristics may or may not be desirable for a creative writing assistant.

Evaluating an Inference Strategy for a Creative Writing Assistant

A large language model is tasked with completing a sentence. It employs a strategy where, at each step, it selects the single most probable word to add next, without considering the impact of this choice on future word selections. Analyze the primary advantage and the main potential disadvantage of this strategy for generating high-quality, coherent text. Illustrate the disadvantage with a brief, hypothetical example.

Analyzing a Step-by-Step Text Generation Strategy

A greedy search sequence is not always optimal because early sub-optimal token choices can lead to higher subsequent probabilities. Suppose a greedy search yields the sequence "A", "B", "C", "<eos>" with a total probability of $$0.048$$. Alternatively, if the model instead selects the token with the second highest conditional probability at step 2 (e.g., token "C" with probability $$0.3$$), the context for step 3 changes from "A", "B" to "A", "C". This change alters the conditional probabilities for subsequent tokens. Choosing "B" at step 3 and "<eos>" at step 4 might yield new probabilities, such as $$0.6$$ and $$0.6$$, respectively. The overall probability of this new sequence "A", "C", "B", "<eos>" would be $$0.5 \times 0.3 \times 0.6 \times 0.6 = 0.054$$, which is greater than the greedy search's result of $$0.048$$, proving the greedy approach missed the most likely sequence.

Example of a Suboptimal Greedy Search Outcome

A language model is generating a response, starting with the phrase 'The best way to learn is...'. The model must decide on the next three words. At each step, it uses a strategy where it always chooses the single word with the highest probability. Given the probabilities below, which sequence of three words will the model generate?

**Step 1 (after '...is'):**
- `by`: 0.5
- `through`: 0.3
- `with`: 0.2

**Step 2 (if 'by' is chosen):**
- `doing`: 0.6
- `reading`: 0.3
- `listening`: 0.1

**Step 3

Learn Before

Related