Learn Before
Search for Optimal or Sub-optimal Sequences in LLM Inference
The search phase in LLM inference involves finding an output sequence y that is considered optimal or sub-optimal. The quality of any candidate sequence is determined by its conditional log-probability, log Pr(y|x), which the search process aims to maximize.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Hypothesis in LLM Inference
Mathematical Formulation of the Search Problem in LLM Inference
Exploration vs. Exploitation in LLM Search
Search Tree Structure in Token Generation
Heuristic Search Algorithms for LLM Inference
Efficient Generation of Candidate Solutions via Search Algorithms
Search for Optimal or Sub-optimal Sequences in LLM Inference
Root of the Search Space as a Representation of Input (x)
A text generation model has a vocabulary of 10,000 possible words it can choose from for each position in a sequence. If this model were to find the optimal output by evaluating every single possible sequence, how would the total number of sequences to check change if the desired output length is increased from 3 words to 5 words?
Evaluating an Inference Strategy
The Impracticality of Exhaustive Search
Historical Context and Computational Challenges of Maximum Probability Prediction
Mathematical Representation of an Output Sequence
Learn After
A language model is generating text with the goal of producing the most probable output sequence. It has already generated the phrase 'The best way to learn is by...' and must now decide the next word. The model calculates the following probabilities for the next possible words:
Pr('doing' | 'The best way to learn is by...') = 0.6Pr('reading' | 'The best way to learn is by...') = 0.3Pr('sleeping' | 'The best way to learn is by...') = 0.09Pr('car' | 'The best way to learn is by...') = 0.01
To continue constructing the sequence with the highest possible overall probability, which word should the search process select at this step?
Limitations of Probability Maximization in Text Generation
Evaluating Candidate Sequences in LLM Inference