Based on the scenario, explain the fundamental reason why the ratio term is necessary to accurately evaluate the new policy, $\pi_{\theta}$, using only the data collected under the original policy, $\pi_{\text{ref}}$.

Google

In reinforcement learning, a surrogate objective is an alternative objective function that is optimized in place of the true performance function. A common example is the off-policy objective, which uses importance sampling to evaluate the current policy with data from a reference policy: $ \mathbb{E}_{\tau \sim \pi_{\theta_{\text{ref}}}} \left[ \frac{\text{Pr}_{\theta}(\tau)}{\text{Pr}_{\theta_{\text{ref}}}(\tau)} R(\tau) \right] $. This formulation acts as a proxy, or surrogate, for the true on-policy objective $ \mathbb{E}_{\tau \sim \pi_{\theta}} [R(\tau)] $.

Surrogate Objective in Reinforcement Learning

The surrogate objective function, which evaluates a policy $$\pi_\theta$$ using trajectories sampled from a reference policy $$\pi_{\theta_{\mathrm{ref}}}$$, is mathematically equivalent to the true on-policy objective. This equivalence is established by expanding the expectation of the importance-sampled reward into its summation form, where the probability of a trajectory under the reference policy cancels out: $$ \mathbb{E}_{\tau \sim \pi_{\theta_{\mathrm{ref}}}} \Big[ \frac{\mathrm{Pr}_{\theta}(\tau)}{\mathrm{Pr}_{\theta_{\mathrm{ref}}}(\tau)} R(\tau) \Big] = \sum_{\tau} \mathrm{Pr}_{\theta_{\mathrm{ref}}}(\tau) \frac{\mathrm{Pr}_{\theta}(\tau)}{\mathrm{Pr}_{\theta_{\mathrm{ref}}}(\tau)} R(\tau) = \sum_{\tau} \mathrm{Pr}_{\theta}(\tau) R(\tau) = \mathbb{E}_{\tau \sim \pi_{\theta}} [R(\tau)] $$. However, this strict mathematical equivalence holds only when the expectation is performed over the entire sequence space. In practice, because policy learning models often sample a relatively small subset of sequences, the sampling methodology itself significantly influences the resulting estimates.

Equivalence of the Surrogate Objective and the On-Policy Objective

When the current policy parameters are identical to the reference policy parameters, a condition denoted by $\theta = \theta_{\text{ref}}$, the standard importance-sampled surrogate objective simplifies. The importance sampling ratio becomes one, causing the surrogate objective's value to equal the expected reward of the reference policy: $$ \left. \mathbb{E}_{\tau \sim \pi_{\theta_{\text{ref}}}} \left[ \frac{\text{Pr}_{\theta}(\tau)}{\text{Pr}_{\theta_{\text{ref}}}(	au)} R(\tau) \right] \right|_{\theta=\theta_{\text{ref}}} = \mathbb{E}_{\tau \sim \pi_{\theta_{\text{ref}}}} [R(\tau)] $$ In this specific context, the term 'surrogate objective' may refer to this simplified expression, which is equivalent to the true on-policy objective at this point.

Surrogate Objective at the Policy Reference Point

A key property of the importance-sampled surrogate objective is that its gradient, when evaluated at the reference policy parameters ($\theta = \theta_{\text{ref}}$), is identical to the standard on-policy policy gradient. This is demonstrated by moving the derivative inside the expectation, which is valid since the sampling distribution $\pi_{\theta_{\text{ref}}}$ does not depend on $\theta$: $$ \frac{\partial}{\partial \theta} \left. \mathbb{E}_{\tau \sim \pi_{\theta_{\text{ref}}}} \left[ \frac{\text{Pr}_{\theta}(\tau)}{\text{Pr}_{\theta_{\text{ref}}}(\tau)} R(\tau) \right] \right|_{\theta=\theta_{\text{ref}}} = \mathbb{E}_{\tau \sim \pi_{\theta_{\text{ref}}}} \left[ \frac{\frac{\partial}{\partial \theta} \text{Pr}_{\theta}(\tau)|_{\theta=\theta_{\text{ref}}}}{\text{Pr}_{\theta_{\text{ref}}}(\tau)} R(\tau) \right] $$ The term inside the expectation on the right, $\frac{\nabla_{\theta} \text{Pr}_{\theta}(\tau)}{\text{Pr}_{\theta_{\text{ref}}}(\tau)}$, when evaluated at $\theta=\theta_{\text{ref}}$, is equivalent to $\nabla_{\theta} \log \text{Pr}_{\theta}(\tau)|_{\theta=\theta_{\text{ref}}}$. Thus, the right-hand side is the definition of the on-policy policy gradient evaluated at the reference policy. This equivalence ensures that at the beginning of an optimization step, the update direction for the surrogate objective is the same as the true policy gradient.

Equivalence of Surrogate and On-Policy Gradients at the Reference Point

Training a Policy with Off-Distribution Data

A reinforcement learning agent is being updated. The current policy is denoted by $\pi_{\theta}$, and a batch of trajectory data has been collected using a previous, fixed policy, $\pi_{\theta_{\text{ref}}}$. To improve the current policy using this existing data, the following objective function is optimized: $L(\theta) = \mathbb{E}_{\tau \sim \pi_{\theta_{\text{ref}}}} \left[ \frac{\text{Pr}_{\theta}(\tau)}{\text{Pr}_{\theta_{\text{ref}}}(\tau)} R(\tau) \right]$. Which statement best analyzes the role of this objective function in the training process?

In the context of policy optimization, an agent's performance is ultimately measured by the on-policy objective, $ \mathbb{E}_{\tau \sim \pi_{\theta}} [R(\tau)] $. However, many algorithms instead optimize a surrogate objective, such as $ \mathbb{E}_{\tau \sim \pi_{\theta_{\text{ref}}}} \left[ \frac{\text{Pr}_{\theta}(\tau)}{\text{Pr}_{\theta_{\text{ref}}}(\tau)} R(\tau) \right] $, where data is sampled from a reference policy $\pi_{\theta_{\text{ref}}}$. Analyze the primary practical advantage of optimizing this surrogate objective compared to directly optimizing the on-policy objective.

Rationale for Using a Surrogate Objective

The formulation of the surrogate objective allows for a functional separation between the sequence sampling process and the reward computation. By utilizing a baseline policy (parameterized by $$\theta_{\mathrm{ref}}$$) to sample a batch of sequences, and then applying the target policy (parameterized by $$\theta$$) to compute the expected reward, the overall procedure avoids the need to sample directly from the policy being evaluated. This decoupling is especially beneficial in reinforcement learning scenarios where generating trajectories from the target policy is computationally expensive or difficult.

Separation of Sampling and Reward Computation in Policy Learning

A primary challenge when utilizing an unclipped surrogate objective function for policy optimization is that the variance in its resulting gradient estimates is typically quite high. This elevated variance introduces significant noise into the parameter updates, which can destabilize the overall learning process and hinder reliable convergence.

Variance in Surrogate Objective Gradient Estimates

To address the high variance and resultant instability in policy gradient estimates, a clipped surrogate objective function is widely used. This objective incorporates a clipping mechanism to bound the importance weights, ensuring that individual policy updates do not become excessively large. The clipped utility function is formally defined as: $$U_{\mathrm{clip}}(\tau;\theta) = \sum_{t=1}^{T} \mathrm{Clip}\Big( \frac{\pi_{\theta}(a_t|s_t)}{\pi_{\theta_{\mathrm{ref}}}(a_t|s_t)} \Big) A(s_t,a_t)$$ where the clipping operation restricts the probability ratio using a specified boundary hyperparameter $$\epsilon$$: $$\mathrm{Clip}\Big( \frac{\pi_{\theta}(a_t|s_t)}{\pi_{\theta_{\mathrm{ref}}}(a_t|s_t)} \Big) = \min\Big( \frac{\pi_{\theta}(a_t|s_t)}{\pi_{\theta_{\mathrm{ref}}}(a_t|s_t)},\mathrm{bound} \big(\frac{\pi_{\theta}(a_t|s_t)}{\pi_{\theta_{\mathrm{ref}}}(a_t|s_t)}, 1 - \epsilon, 1 + \epsilon \big) \Big)$$.

Learn Before

Related