An engineer training an agent with a policy gradient method notices that the learning process is unstable due to high variance in the gradient estimates. To address this, they introduce a baseline which is subtracted from the rewards. What is the expected statistical consequence of this modification?

Differential Impact of Baselines on Variance

Introducing a baseline into a policy gradient algorithm is an effective technique for reducing the variance of the total rewards collected by the agent, which in turn stabilizes the learning process.