Learn Before
Analysis of the Baseline's Effect on Policy Gradient Expectation
In policy gradient methods, a baseline term, which often depends only on the state, is subtracted from the reward to reduce the variance of the gradient estimate. Explain mathematically why the expectation of the gradient contribution from such a baseline is zero, and therefore does not introduce bias to the overall policy gradient estimate. Your explanation should focus on the properties of the score function (the gradient of the log-policy).
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Analysis of the Baseline's Effect on Policy Gradient Expectation
In a policy gradient algorithm, a common technique to stabilize learning is to subtract a calculated value from the total reward of each trajectory before computing the update. This is done to reduce the variability of the updates without altering their expected direction. Which of the following calculated values, if subtracted from the total reward, would introduce an incorrect bias and potentially lead the policy updates in the wrong direction on average?
In the mathematical proof demonstrating that a state-dependent baseline
b(s_t)does not introduce bias to the policy gradient estimate, the expected value of the baseline-related term,E[ (∇θ log πθ(a_t|s_t)) * b(s_t) ], evaluates to zero. Which of the following is the fundamental reason for this outcome?