logo
How it worksCoursesResearch CommunitiesBenefitsAbout Us
Schedule Demo
Learn Before

  • Search (Decoding) Algorithms for LLM Inference

Stopping Criteria in LLM Inference

Stopping criteria are essential rules within LLM inference that determine when the text generation process should conclude. These conditions are necessary to signal the end of decoding, prevent indefinite output, and manage practical considerations like decoding cost and verbosity by avoiding overly long sequences.

0

1

7 months ago

Contributors are:

Gemini AI
Gemini AI
🏆 9

Who are from:

Google
Google
🏆 9

References

Tags

Ch.5 Inference - Foundations of Large Language Models

Foundations of Large Language Models

Foundations of Large Language Models Course

Computing Sciences

Related

  • Heuristic Search Algorithms for LLM Inference

  • Stopping Criteria in LLM Inference

  • Computational Infeasibility of Exhaustive Search in LLM Decoding

  • A language model is given the prompt 'The capital of France is'. Internally, the model's calculations show that the single most probable next word is 'Paris'. However, the model ultimately generates the sequence 'The capital of France is a beautiful city'. Which statement best analyzes the reason for this discrepancy?

  • The Challenge of Generating Optimal Text

  • Analyzing Text Generation Behavior

Learn After

  • End-of-Sequence (EOS) Token as a Stopping Criterion

  • Sequence Count as a Stopping Criterion in Beam Search

  • Maximum Output Length as a Stopping Criterion

  • Cost-Based Stopping Criteria

  • Behavior-Based Stopping Criteria

  • Debugging an Uncontrolled Text Generation System

  • A developer is testing a new text-generation system. They find that when prompted, the system produces a relevant initial response but then continues to generate a long, rambling stream of unrelated text until it is manually interrupted. What is the most fundamental problem with the system's configuration that leads to this behavior?

  • Consequences of Unbounded Text Generation