Analyzing a Flawed Policy Gradient Derivation
A student is attempting to derive the policy gradient objective function. Their derivation is shown below. Identify the specific mathematical error in their steps and explain why this error introduces a fundamental problem that the standard derivation avoids.
Derivation Steps:
- Objective Function:
- Gradient Calculation:
- Conclusion: The student stops here, concluding that for this gradient to be useful, the reward function must be differentiable with respect to the policy parameters .
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Policy Gradient Theorem
Advantage of Policy Gradients: Non-Differentiable Reward Functions
Decomposition of the Trajectory Log-Probability Gradient
Policy Gradient Objective with Advantage Function
Policy Gradient Estimate under Uniform Trajectory Probability
Score Function in Policy Gradients
During the derivation of the policy performance gradient, a key step transforms the expression
Σ [∂Pr_θ(τ)/∂θ] R(τ)into a form that includes the term∂log Pr_θ(τ)/∂θ. What is the primary analytical purpose of this transformation?The following equations represent key steps in deriving the policy gradient. Arrange them in the correct logical order, starting from the initial gradient of the objective function to its final form as an expectation. Note: J(θ) is the objective function, Pr_θ(τ) is the probability of a trajectory τ under policy parameters θ, and R(τ) is the reward for that trajectory.
Analyzing a Flawed Policy Gradient Derivation