Mathematical Justification for Greedy Search

Construction of the Optimal Sequence in Greedy Search

Candidate Set in Greedy Search

A language model is generating a two-token sequence. At the first step, it calculates the probability for the next token: 'Token A' has a probability of 0.6, and 'Token B' has a probability of 0.4. If the model chooses 'Token A', the most probable subsequent token is 'Token C' (with a conditional probability of 0.5). If the model had chosen 'Token B', the most probable subsequent token would be 'Token D' (with a conditional probability of 0.9). A text generation algorithm is used that, at every step, commits to the single token with the highest immediate probability. Based on this process, which sequence will be generated and why?

Algorithm Suitability for Text Generation Tasks

When generating a sequence of text, an algorithm that selects the single most probable token at each step is guaranteed to produce the overall most probable sequence.

Analyzing Suboptimal Outcomes in Text Generation

Selecting and Justifying a Decoding Policy for Two Production Use Cases

Debugging Decoding: Balancing Determinism, Diversity, and Length in a Regulated Product

Post-incident analysis: fixing repetition and truncation by tuning decoding

Choosing a Decoding Configuration Under Latency, Diversity, and Length Constraints

Release-readiness decision: decoding configuration for a customer-facing summarization feature

Decoding policy decision for a multilingual support assistant under safety, latency, and verbosity constraints

You are tuning decoding for an internal "meeting-n...

You’re implementing an LLM feature that generates ...

You’re building an internal “RFP response drafter”...

You’re deploying an LLM to draft customer-facing i...

Beam search