Policy Optimization Derivation Step
A key insight in policy optimization is that minimizing a specific KL divergence is equivalent to maximizing a reward-based objective. Consider the expression for the KL divergence between a learned policy, π_θ(·|x), and an optimal policy, π*(·|x):
KL(π_θ(·|x) || π*(·|x))
Given the definition of the optimal policy as π*(y|x) = (1/Z(x)) * π_ref(y|x) * exp(r(x,y)/β), expand the KL divergence expression. Your final answer should be in terms of the learned policy π_θ, the reference policy π_ref, the reward r(x,y), the scaling factor β, and the normalization term Z(x). Do not drop any terms.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Simplified Policy Optimization Objective as KL Divergence Minimization
In a derivation showing that a policy optimization objective is equivalent to minimizing KL divergence, the objective is simplified from
arg min_θ E_{x~D} [ KL(π_θ(·|x) || π*(·|x)) - log Z(x) ]toarg min_θ E_{x~D} [ KL(π_θ(·|x) || π*(·|x)) ]. Why is it valid to remove thelog Z(x)term during this final simplification?A policy optimization objective can be shown to be equivalent to minimizing a KL divergence. Arrange the following expressions to show the correct logical sequence of this mathematical derivation, starting from the point where the optimal policy has been substituted into the objective.
Policy Optimization Derivation Step