Diagnosing Training Instability
Given the following scenario, explain why the chosen baseline is suboptimal and is likely contributing to training instability. Then, propose a more effective constant value for the baseline and justify why it would lead to more stable gradient updates.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
In a policy gradient algorithm, the update for the policy parameters is influenced by the term
(R - b), whereRis the total reward for an episode andbis a baseline. Imagine you are training an agent where most episodes yield a small, positive total reward (e.g., between 1 and 5). If you set the baselinebto a constant, large positive value (e.g., 10), what is the most likely consequence for the learning process?Diagnosing Training Instability
In policy gradient methods, subtracting a baseline from the total reward is a technique used to reduce gradient variance. A key property of a properly chosen baseline is that it alters the expected value of the policy gradient, making the updates more conservative.