Policy Gradient with Reward-to-Go and Baseline

In a reinforcement learning setting, the update for a policy π_θ(a_t|s_t) depends on a gradient calculation. For a specific timestep t within a sequence of actions, the term influencing the update can be structured as follows, separating rewards that occurred before time t from those that occurred at or after time t:

log π_θ(a_t|s_t) * ( (∑_{k=1}^{t-1} r_k) + (∑_{k=t}^{T} r_k) )

When computing the gradient of this expression with respect to the policy parameters θ, how does the ∑_{k=1}^{t-1} r_k term (the sum of past rewards) influence the gradient associated with the action at time t?

Policy Gradient Optimization

Consider the calculation of the policy gradient with respect to the parameters θ for an action a_t taken at time t. A proposed update rule multiplies the term ∇_θ log π_θ(a_t|s_t) by the sum of all rewards in the trajectory (∑_{k=1}^{T} r_k). Including the rewards received before time t (∑_{k=1}^{t-1} r_k) in this multiplication introduces a systematic error, or bias, into the resulting gradient estimate.

Policy Gradient with Reward-to-Go and Baseline

Calculating Advantage from a Trajectory

In the context of estimating the advantage of taking an action a_t in a state s_t, the formula A(s_t, a_t) = (∑_{k=t}^{T} r_k) - V(s_t) is often used. What is the primary role of the reward-to-go term, ∑_{k=t}^{T} r_k, within this specific estimation?

In a given trajectory, if the calculated advantage A(s_t, a_t) = (∑_{k=t}^{T} r_k) - V(s_t) is negative, it implies that the action a_t taken in state s_t led to a sequence of rewards that was worse than the average expected outcome from that state.

Policy Gradient with Reward-to-Go and Baseline

In a method for training a decision-making agent, an update rule is derived. Consider the following intermediate expression used to calculate the gradient for a single trajectory of states, actions, and rewards:

$\nabla_\theta J(\theta) \propto \sum_{t=1}^{T} \left[ \nabla_\theta \log \pi_\theta(a_t|s_t) \cdot \left( \left( \sum_{k=1}^{t-1} r_k \right) + \left( \sum_{k=t}^{T} r_k \right) - b \right) \right]$

Here, t is a specific timestep within the trajectory of length T, \pi_\theta(a_t|s_t) is the probability of taking action a_t in state s_t, r_k is the reward at timestep k, and b is a constant value. Which statement best analyzes the relationship between the policy term for timestep t ( \nabla_\theta \log \pi_\theta(a_t|s_t) ) and the two components of the reward sum?

In the context of improving a policy gradient estimator, the total reward for a trajectory, ( \sum_{k=1}^{T} r_k ), is often rewritten inside the gradient calculation for a specific timestep t as ( \sum_{k=1}^{t-1} r_k + \sum_{k=t}^{T} r_k ). This specific algebraic decomposition, by itself, alters the expected value of the gradient estimate.

Rationale for Reward Decomposition