Policy Gradient with Reward-to-Go and Baseline

In a method for training a decision-making agent, an update rule is derived. Consider the following intermediate expression used to calculate the gradient for a single trajectory of states, actions, and rewards:

$\nabla_\theta J(\theta) \propto \sum_{t=1}^{T} \left[ \nabla_\theta \log \pi_\theta(a_t|s_t) \cdot \left( \left( \sum_{k=1}^{t-1} r_k \right) + \left( \sum_{k=t}^{T} r_k \right) - b \right) \right]$

Here, t is a specific timestep within the trajectory of length T, `\pi_\theta

In the context of improving a policy gradient estimator, the total reward for a trajectory, $\sum_{k=1}^{T} r_k$, is often rewritten inside the gradient calculation for a specific timestep t as $\sum_{k=1}^{t-1} r_k + \sum_{k=t}^{T} r_k$. This specific algebraic decomposition, by itself, alters the expected value of the gradient estimate.

Rationale for Reward Decomposition