Calculating Trajectory Utility with Importance Sampling
An agent's policy is being updated. The utility of a trajectory is calculated by re-weighting the advantage of each action. The formula for the utility U of a two-step trajectory is: Based on the data in the case study below, calculate the total utility U for this trajectory and explain what the final value implies for the policy update.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Clipped Utility Function with Upper-Bound Clipping
An agent's policy is being updated using an objective function that relies on importance sampling. Consider a single time step
tin a trajectory where the calculated advantageA(s_t, a_t)is large and positive. At the same time, the importance sampling ratioπ_θ(a_t|s_t) / π_θ_ref(a_t|s_t)is also large (e.g., 5.0), indicating the current policy is much more likely to choose actiona_tthan the reference policy was.Given the objective function
U(τ; θ) = Σ [π_θ(a_t|s_t) / π_θ_ref(a_t|s_t)] * A(s_t, a_t), what is the most direct consequence of this situation for this specific time step's contribution to the policy update?Calculating Trajectory Utility with Importance Sampling
In the context of updating a policy using an objective function with importance sampling, if the ratio of the current policy's probability to the reference policy's probability for a given action is greater than 1, this will always increase the likelihood of that action being selected in the subsequent policy update.