Temperature-Scaled Softmax for Token Probability
This formula calculates the probability of generating a specific token given an input and the preceding tokens . It applies the softmax function to the model's output scores (logits), denoted by , for all possible tokens in the vocabulary . A temperature parameter, , scales the scores before the exponential function is applied. This scaling adjusts the shape of the probability distribution: lower temperatures result in a sharper, more deterministic distribution, while higher temperatures create a flatter, more random distribution. The formula is: .
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Token Sampling from a Conditional Probability Distribution
Calculating Next-Token Probability
An autoregressive model is generating a sequence and has computed the following unnormalized scores (logits) for three candidate next tokens: Token A (3.0), Token B (1.0), and Token C (0.0). If a constant value of 10.0 is added to each of these three logits before the final probability normalization step, how will the resulting conditional probabilities for the tokens be affected?
An autoregressive language model calculates unnormalized scores (logits) for a set of candidate next tokens. These scores are then transformed into a probability distribution. What is the primary reason for applying an exponential function to each logit before the final normalization step?
Temperature-Scaled Softmax for Token Probability
Temperature-Scaled Softmax for Renormalized Probability
Token Sampling from a Conditional Probability Distribution
Temperature-Scaled Softmax for Renormalized Probability
A language model has calculated the following raw scores (logits) for the next potential token:
{'mat': 3.0, 'rug': 2.5, 'chair': 2.0, 'moon': -1.0}. To control the randomness of the output, a temperature parameter is applied to these scores before they are converted into a final probability distribution for sampling. Which of the following probability distributions most likely resulted from applying a low temperature (e.g., a value less than 1.0)?Troubleshooting a Factual Chatbot's Output
You are configuring a text generation model for different tasks. Match each task with the description of the temperature setting that would be most appropriate to achieve the desired output.
Temperature-Scaled Softmax for Token Probability
Learn After
A language model is tasked with completing the sentence: 'The old sea captain stared at the stormy sky and said, 'It's going to be a...'' The model's internal scores (logits) for the next token are highest for 'rough', followed by 'long', 'dark', and then 'whale'. The model generates two different completions using different settings:
- Completion A: '...rough night.'
- Completion B: '...whale of a tale.'
Based on the probability formula , which statement most accurately analyzes the relationship between the temperature parameter () and the generated completions?
Effect of Temperature on Token Generation
Analyzing Temperature's Impact on Token Probabilities