Learn Before
Using Temperature with Softmax to Control Randomness in Token Selection
The randomness of token selection in large language models can be finely controlled by applying a temperature parameter, , to the Softmax function, which adjusts the sharpness of the probability distribution derived from the raw logits. A higher temperature value diminishes the differences between logits, making the probability distribution more uniform and giving all candidate tokens a more equal chance of being selected, thereby increasing the diversity of the generated output. Conversely, setting the temperature to a lower value sharpens the distribution, increasing the likelihood of selecting high-probability tokens and leading to more deterministic outputs. For instance, setting the Top-p threshold to 1 and the temperature close to zero makes the sampling process equivalent to a greedy search.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Next Token Prediction Task
Token Sampling from a Conditional Probability Distribution
Using Temperature with Softmax to Control Randomness in Token Selection
A language model is generating text and has produced the sequence 'The sky is'. It then calculates the following probability distribution for the next potential token:
{'blue': 0.75, 'green': 0.15, 'bright': 0.08, 'falling': 0.02}. If the model is configured to always select the single token with the highest probability, which token will it choose next?Analyzing Token Selection Strategies
A language model is generating text and encounters the same input sequence on two separate occasions, producing two different probability distributions for the next token, shown below.
- Distribution A:
{'meal': 0.90, 'dish': 0.05, 'surprise': 0.03, 'error': 0.02} - Distribution B:
{'soup': 0.30, 'stew': 0.25, 'salad': 0.22, 'dessert': 0.23}
Which of the following statements provides the most accurate analysis of these two distributions regarding the token selection process?
- Distribution A:
To ensure the generated text is as coherent and factually accurate as possible, a language model must always select the single token with the highest probability from the distribution at each step of the generation process.
Learn After
Token Sampling from a Conditional Probability Distribution
Temperature-Scaled Softmax for Renormalized Probability
A language model has calculated the following raw scores (logits) for the next potential token:
{'mat': 3.0, 'rug': 2.5, 'chair': 2.0, 'moon': -1.0}. To control the randomness of the output, a temperature parameter is applied to these scores before they are converted into a final probability distribution for sampling. Which of the following probability distributions most likely resulted from applying a low temperature (e.g., a value less than 1.0)?Troubleshooting a Factual Chatbot's Output
You are configuring a text generation model for different tasks. Match each task with the description of the temperature setting that would be most appropriate to achieve the desired output.
Temperature-Scaled Softmax for Token Probability