In a policy gradient algorithm, the update for the policy parameters is influenced by the term (R - b), where R is the total reward for an episode and b is a baseline. Imagine you are training an agent where most episodes yield a small, positive total reward (e.g., between 1 and 5). If you set the baseline b to a constant, large positive value (e.g., 10), what is the most likely consequence for the learning process?

Diagnosing Training Instability

In policy gradient methods, subtracting a baseline from the total reward is a technique used to reduce gradient variance. A key property of a properly chosen baseline is that it alters the expected value of the policy gradient, making the updates more conservative.