Temperature-Scaled Softmax for Token Probability
This formula calculates the probability of generating a specific token given an input and the preceding tokens . It applies the softmax function to the model's output scores (logits), denoted by , for all possible tokens in the vocabulary . A temperature parameter, , scales the scores before the exponential function is applied. This scaling adjusts the shape of the probability distribution: lower temperatures result in a sharper, more deterministic distribution, while higher temperatures create a flatter, more random distribution. The formula is: .
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Next-Word Prediction Model
A language model's prediction rule is to select the output with the highest conditional probability. Given the input text 'The ocean is deep and...', the model computes the following probabilities for the next word:
- P('mysterious' | 'The ocean is deep and...') = 0.55
- P('blue' | 'The ocean is deep and...') = 0.30
- P('empty' | 'The ocean is deep and...') = 0.10
- P('loud' | 'The ocean is deep and...') = 0.05
Based on its prediction rule, which word will the model choose?
Temperature-Scaled Softmax for Token Probability
Evaluating the 'Arg Max' Prediction Strategy
Learn After
A language model is tasked with completing the sentence: 'The old sea captain stared at the stormy sky and said, 'It's going to be a...'' The model's internal scores (logits) for the next token are highest for 'rough', followed by 'long', 'dark', and then 'whale'. The model generates two different completions using different settings:
- Completion A: '...rough night.'
- Completion B: '...whale of a tale.'
Based on the probability formula , which statement most accurately analyzes the relationship between the temperature parameter () and the generated completions?
Effect of Temperature on Token Generation
Analyzing Temperature's Impact on Token Probabilities