Reward-Weighted Probability Distribution

Consider a scenario where for a given input $\mathbf{x}$, there are only two possible outputs, $\mathbf{y}_1$ and $\mathbf{y}_2$. A reference model assigns probabilities $\pi_{\text{ref}}(\mathbf{y}_1|\mathbf{x}) = 0.6$ and $\pi_{\text{ref}}(\mathbf{y}_2|\mathbf{x}) = 0.4$. A reward function gives scores $r(\mathbf{x}, \mathbf{y}_1) = 2$ and $r(\mathbf{x}, \mathbf{y}_2) = 1$. Assuming the scaling factor $\beta$ is 1, what is the value of the normalization factor $Z(\mathbf{x})$, which is calculated as $Z(\mathbf{x}) = \sum_{\mathbf{y}} \pi_{\text{ref}}(\mathbf{y}|\mathbf{x}) \exp(r(\mathbf{x}, \mathbf{y}))$?

Consider the calculation of a normalization factor using the formula: $Z(\mathbf{x}) = \sum_{\mathbf{y}} \pi_{\theta_{\text{ref}}}(\mathbf{y}|\mathbf{x}) \exp \left(\frac{1}{\beta}r(\mathbf{x}, \mathbf{y})\right)$ If the reward function $r(\mathbf{x}, \mathbf{y})$ consistently returns a value of 0 for all possible outputs $\mathbf{y}$, the normalization factor $Z(\mathbf{x})$ will always be equal to 1.

Impact of Scaling Factor on Normalization

An AI text generation system adjusts the likelihood of different outputs using the formula: New_Likelihood = Base_Likelihood * exp((1/β) * Reward). In this formula, 'Base_Likelihood' is the initial probability from a reference model, 'Reward' is a score for the output's quality, and 'β' is a positive 'temperature' parameter. A team wants to use this system to generate a diverse set of creative, high-quality story endings. They are comparing two settings for the temperature parameter: β = 0.5 and β = 2.0. Which setting should they choose to better achieve their goal, and why?

Tuning a Generative Model for Different Tasks

Effect of Temperature Scaling on a Reward-Modified Distribution

Reward-Weighted Probability Distribution