Policy Gradient with Reward-to-Go and Baseline

Calculating Advantage from a Trajectory

In the context of estimating the advantage of taking an action a_t in a state s_t, the formula A(s_t, a_t) = (∑_{k=t}^{T} r_k) - V(s_t) is often used. What is the primary role of the reward-to-go term, ∑_{k=t}^{T} r_k, within this specific estimation?

In a given trajectory, if the calculated advantage A(s_t, a_t) = (∑_{k=t}^{T} r_k) - V(s_t) is negative, it implies that the action a_t taken in state s_t led to a sequence of rewards that was worse than the average expected outcome from that state.