Value Function Loss Minimization in RLHF

PPO Objective Formula for LLM Training in RLHF

During the final training phase of a language model guided by human feedback, both a policy (the language model itself) and a value function are updated in tandem. Which of the following statements best analyzes the distinct roles and update mechanisms of these two components in this joint optimization process?

In the final stage of training a language model with feedback, a policy and a value function are optimized concurrently. Match each component to its primary optimization objective and its role in this process.

Value Model Update Frequency in RLHF

Advantage Function as TD Error in RLHF

Diagnosing Training Stagnation in Joint Optimization