A sequence generation model is being trained using a policy gradient method. At a specific step t, the model must choose between two possible next tokens: 'innovative' and 'effective'. The model's internal calculations provide the following values for this step:

• For token 'innovative':

  • Log-probability log π(y_t|...): -3.0
  • Advantage A(...): +4.0

• For token 'effective':

  • Log-probability log π(y_t|...): -1.2
  • Advantage A(...): +2.0

Based on the utility function U used in policy gradient methods, which is a sum of log π * A terms over the sequence, which token's selection results in a larger (i.e., less negative) contribution to the total utility U for the entire sequence at this specific step t?

Analyzing Policy Gradient Updates for Text Generation

Consider a sequence generation model being trained with a policy gradient method. If the advantage function A(x, y_<t, y_t) returns a negative value for a specific token y_t at a given step, the training objective will encourage the model to increase the probability of selecting that token y_t in similar future situations.

Basic A2C Formulation for LLMs