Purpose of Reward Decomposition in Policy Gradient
A key step in reformulating the policy gradient expression involves rewriting the total reward for a trajectory, (sum_{k=1 to T} r_k), as a sum of two components: rewards accumulated before a given timestep t (sum_{k=1 to t-1} r_k) and rewards accumulated from that timestep onward (sum_{k=t to T} r_k). Explain the primary motivation for performing this decomposition. What does this separation of rewards allow for in subsequent steps of the gradient calculation?
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Irrelevance of Past Rewards for Policy Gradient Calculation
Consider the following mathematical derivation, which attempts to rewrite the policy gradient with a baseline. The goal is to separate the reward term into components that occurred before and after a specific action. Analyze the steps and identify which one contains a logical or mathematical error.
Derivation: Let the policy gradient objective be:
Purpose of Reward Decomposition in Policy Gradient
A common technique to improve the stability of a policy-based learning algorithm involves rewriting its core update rule. The goal is to isolate the influence of rewards that occur after an action is taken from those that occur before. Below are four key stages of this mathematical derivation. Arrange them in the correct logical order, from the initial formulation to the final decomposed form. (Note: For simplicity, the expectation over trajectories is omitted).