Diagnosing Training Stagnation in Joint Optimization
Analyze the following training scenario. Why might a highly accurate value function fail to produce a corresponding improvement in the policy's performance? In your explanation, distinguish between the optimization goal of the value function and the optimization goal of the policy.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Value Function Loss Minimization in RLHF
PPO Objective Formula for LLM Training in RLHF
During the final training phase of a language model guided by human feedback, both a policy (the language model itself) and a value function are updated in tandem. Which of the following statements best analyzes the distinct roles and update mechanisms of these two components in this joint optimization process?
In the final stage of training a language model with feedback, a policy and a value function are optimized concurrently. Match each component to its primary optimization objective and its role in this process.
Value Model Update Frequency in RLHF
Advantage Function as TD Error in RLHF
Diagnosing Training Stagnation in Joint Optimization