Google

The randomness of token selection in large language models can be finely controlled by applying a temperature parameter, $$\beta$$, to the Softmax function, which adjusts the sharpness of the probability distribution derived from the raw logits. A higher temperature value diminishes the differences between logits, making the probability distribution more uniform and giving all candidate tokens a more equal chance of being selected, thereby increasing the diversity of the generated output. Conversely, setting the temperature to a lower value sharpens the distribution, increasing the likelihood of selecting high-probability tokens and leading to more deterministic outputs. For instance, setting the Top-$$p$$ threshold to $$1$$ and the temperature close to zero makes the sampling process equivalent to a greedy search.

Using Temperature with Softmax to Control Randomness in Token Selection

In autoregressive text generation, after computing the conditional probability distribution for the next token, $\overline{\text{Pr}}(y_i|\mathbf{x}, \mathbf{y}_{<i})$, the next step is to draw a sample from it. This sampling process, which selects a specific token $\bar{y}_i$, is formally expressed as drawing from the distribution: $$\bar{y}_i \sim \overline{\text{Pr}}(y_i|\mathbf{x}, \mathbf{y}_{<i})$$

Token Sampling from a Conditional Probability Distribution

To control the randomness in token selection, the probability distribution can be reshaped using a temperature parameter, $\beta$. The renormalized conditional probability of a token $y_i$, given the context $(\mathbf{x}, \mathbf{y}_{<i})$, is calculated by applying a temperature-scaled Softmax function to its logit, $u_{y_i}$, and normalizing over a restricted set of candidate tokens $\overline{V}_i$. The formula is: $$ \overline{\text{Pr}}(y_i|\mathbf{x}, \mathbf{y}_{<i}) = \frac{\exp(u_{y_i}/\beta)}{\sum_{y_j \in \overline{V}_i} \exp(u_{y_j}/\beta)} $$

Temperature-Scaled Softmax for Renormalized Probability

A language model has calculated the following raw scores (logits) for the next potential token: `{'mat': 3.0, 'rug': 2.5, 'chair': 2.0, 'moon': -1.0}`. To control the randomness of the output, a temperature parameter is applied to these scores before they are converted into a final probability distribution for sampling. Which of the following probability distributions most likely resulted from applying a **low temperature** (e.g., a value less than 1.0)?

A team develops a chatbot intended to provide precise, factual answers for a customer support knowledge base. During testing, they observe that while the chatbot is fluent, it frequently provides answers that are imaginative and sometimes factually incorrect. Upon reviewing the model's text generation configuration, they find the temperature parameter is set to 1.5. Based on this information, analyze the likely cause of the chatbot's undesirable behavior and explain the reasoning behind your conclusion.

Troubleshooting a Factual Chatbot's Output

You are configuring a text generation model for different tasks. Match each task with the description of the temperature setting that would be most appropriate to achieve the desired output.

Learn Before

Related