In a reinforcement learning process, a new policy defined by parameters θ is evaluated using an objective function that relies on data from a reference policy with parameters θ_ref. The objective function is:

J(θ) = E_{τ ~ π_{θ_ref}} [ (Pr_θ(τ) / Pr_{θ_ref}(τ)) * R(τ) ]

Where τ is a trajectory, Pr(τ) is the probability of that trajectory, R(τ) is its total reward, and E_{τ ~ π_{θ_ref}} denotes the expected value over trajectories from the reference policy.

What does this objective functi

Reasoning for Objective Simplification

In a reinforcement learning scenario, the performance of a new policy, defined by parameters θ, is often estimated using an objective function that relies on data collected from a reference policy, defined by parameters θ_ref. This objective function is given by: $J(\theta) = \mathbb{E}_{\tau \sim \pi_{\theta_{\text{ref}}}} \left[ \frac{\text{Pr}_{\theta}(\tau)}{\text{Pr}_{\theta_{\text{ref}}}(\tau)} R(\tau) \right]$ where τ represents a trajectory, Pr(τ) is the probability of that trajector