Learn Before
Stopping Criteria in LLM Inference
Stopping criteria are essential rules within LLM inference that determine when the text generation process should conclude. These conditions are necessary to signal the end of decoding, prevent indefinite output, and manage practical considerations like decoding cost and verbosity by avoiding overly long sequences.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Heuristic Search Algorithms for LLM Inference
Stopping Criteria in LLM Inference
Computational Infeasibility of Exhaustive Search in LLM Decoding
A language model is given the prompt 'The capital of France is'. Internally, the model's calculations show that the single most probable next word is 'Paris'. However, the model ultimately generates the sequence 'The capital of France is a beautiful city'. Which statement best analyzes the reason for this discrepancy?
The Challenge of Generating Optimal Text
Analyzing Text Generation Behavior
Learn After
End-of-Sequence (EOS) Token as a Stopping Criterion
Sequence Count as a Stopping Criterion in Beam Search
Maximum Output Length as a Stopping Criterion
Cost-Based Stopping Criteria
Behavior-Based Stopping Criteria
Debugging an Uncontrolled Text Generation System
A developer is testing a new text-generation system. They find that when prompted, the system produces a relevant initial response but then continues to generate a long, rambling stream of unrelated text until it is manually interrupted. What is the most fundamental problem with the system's configuration that leads to this behavior?
Consequences of Unbounded Text Generation