Derivation of Reward Decomposition in Policy Gradient with Baseline

Unbiased Nature of Policy Gradient with Baseline

In a reinforcement learning task, an agent completes two distinct trajectories. Trajectory A results in a total reward of +20, and Trajectory B results in a total reward of +5. To update the agent's policy, a baseline value of +12 is subtracted from each trajectory's total reward. Based on this information, how will the policy updates derived from these two trajectories differ?

Consider the formula for the policy gradient estimate with a baseline: $\frac{\partial J(\theta)}{\partial \theta} = \frac{1}{|D|} \sum_{\tau \in D} \left( \frac{\partial}{\partial \theta} \sum_{t=1}^{T} \log \pi_{\theta}(a_t|s_t) \right) \left( \sum_{t=1}^{T} r_t - b \right)$ According to this formula, the baseline value b is subtracted from the reward r_t at each individual timestep t within a trajectory to reduce variance.

Stabilizing Policy Gradient Training