Learn Before
Formula for Optimal Output Sequence in LLMs
In language model inference, the optimal output sequence, denoted as , is found by maximizing the conditional log probability of the output sequence given the input sequence . This objective is formally expressed by decomposing the joint probability over the output tokens: In this formulation, the input sequence is represented as , and the equation models the log probability of predicting subsequent tokens starting from position , rather than position .
0
1
Tags
Foundations of Large Language Models
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
A language model's inference process aims to find an output sequence
ythat maximizes the conditional probabilityPr(y|x)given an inputx. Suppose the model has the input 'The sun is shining and the sky is' and calculates the probabilities for the next word as follows:Pr('blue' | 'The sun is shining and the sky is') = 0.65Pr('clear' | 'The sun is shining and the sky is') = 0.25Pr('vast' | 'The sun is shining and the sky is') = 0.09- `Pr('falling' | 'The sun is shining and
A language model's objective is to find the output sequence with the highest overall conditional probability. Given the input 'The weather is', the model needs to generate a two-word sequence. It has calculated the following probabilities:
Probabilities for the first word:
- Pr('nice' | 'The weather is') = 0.6
- Pr('cold' | 'The weather is') = 0.4
Probabilities for the second word, depending on the first:
- Pr('today' | 'The weather is nice') = 0.5
- Pr('and' | 'The weather is cold') = 0.9
Ba
Comparing Output Sequence Probabilities
Formula for Optimal Output Sequence in LLMs