Equivalence of the Surrogate Objective and the On-Policy Objective

Surrogate Objective at the Policy Reference Point

Equivalence of Surrogate and On-Policy Gradients at the Reference Point

Training a Policy with Off-Distribution Data

A reinforcement learning agent is being updated. The current policy is denoted by $\pi_{\theta}$, and a batch of trajectory data has been collected using a previous, fixed policy, $\pi_{\theta_{\text{ref}}}$. To improve the current policy using this existing data, the following objective function is optimized: $L(\theta) = \mathbb{E}_{\tau \sim \pi_{\theta_{\text{ref}}}} \left[ \frac{\text{Pr}_{\theta}(\tau)}{\text{Pr}_{\theta_{\text{ref}}}(\tau)} R(\tau) \right]$. Which statement best analyzes the role of this objective function in the training process?

Rationale for Using a Surrogate Objective

Separation of Sampling and Reward Computation in Policy Learning

Variance in Surrogate Objective Gradient Estimates

Clipped Surrogate Objective Function