Google

The gradient of the policy performance objective, $$J(\theta)$$, with respect to the policy parameters $$\theta$$ is derived using the log-derivative trick. This mathematical technique transforms the derivative into an expectation that can be estimated from sampled trajectories. The derivation is as follows: $$ \frac{\partial J(\theta)}{\partial \theta} = \frac{\partial}{\partial \theta} \sum_{\tau \in \mathcal{D}} \mathrm{Pr}_{\theta}(\tau)R(\tau) $$ $$ = \sum_{\tau \in \mathcal{D}} \frac{\partial \mathrm{Pr}_{\theta}(\tau)}{\partial \theta} R(\tau) $$ By multiplying and dividing by $$\mathrm{Pr}_{\theta}(\tau)$$, we can introduce the gradient of the logarithm: $$ = \sum_{\tau \in \mathcal{D}} \mathrm{Pr}_{\theta}(\tau) \frac{1}{\mathrm{Pr}_{\theta}(\tau)} \frac{\partial \mathrm{Pr}_{\theta}(\tau)}{\partial \theta} R(\tau) $$ $$ = \sum_{\tau \in \mathcal{D}} \mathrm{Pr}_{\theta}(\tau) \frac{\partial \log \mathrm{Pr}_{\theta}(\tau)}{\partial \theta} R(\tau) $$ This final form shows that the policy gradient is the expected value of the score function, $$\frac{\partial \log \mathrm{Pr}_{\theta}(\tau)}{\partial \theta}$$, weighted by the cumulative reward, $$R(\tau)$$.

Derivation of the Policy Gradient Objective Function

We want to maximize the rewards based on parameters $\theta$, so we can use the expected return of policy π starting from a given state $s_0$ as the objective function:
$$J(\theta) = \sum_s \rho^{\pi}(s) \sum_a \pi(s,a) R'(s,a) $$
where $R'(s,a) = \sum_{s'\in S} T(s,a,s') R(s,a,s')$ and $\rho^{\pi}(s)$ is the discounted state distribution ((meaning the probability of being at state s when following policy 𝜋))
$$ \rho^{\pi}(s) = \sum_{t=0}^{\infty} \gamma^t P(s_t=s|s_0, \pi) $$

According to Policy Gradient Theorem,
$$\nabla_{\theta} J(\theta) \propto  \sum_s \rho^{\pi}(s) \sum_a \nabla_{\theta} \pi(s,a;\theta) Q(s,a;\pi_{\theta}) $$

$\nabla_{\theta} \pi(s,a;\theta) $ is the gradient of 𝜋 given s and 𝜽 and the simplest way to estimate it is to use a score function gradient estimator because
$$\nabla_{\theta} \pi(s,a;\theta) = \pi (s,a;\theta) \frac{\nabla_{\theta}\pi (s,a;\theta)}{\pi (s,a;\theta)} = \pi (s,a;\theta) \nabla_{\theta} \log(\pi(s,a;\theta))$$

$ Q(s,a;\pi_{\theta})$ is the action value function under 𝜋, the simplest way to estimate it is use the cumulative return from entire trajectories.


Policy Gradient Theorem

A significant advantage of the policy gradient method is that the cumulative reward function, R(τ), is not required to be differentiable. This is because the gradient is calculated with respect to the logarithm of the policy's probability, not the reward itself. This property allows for the application of any type of reward function in reinforcement learning, including those that are discontinuous or arbitrarily complex.

Advantage of Policy Gradients: Non-Differentiable Reward Functions

By treating sequence generation as a Markov decision process, the gradient of the log-probability of a trajectory $$\tau$$ with respect to policy parameters $$\theta$$ can be decomposed into two distinct components: the policy gradient and the dynamics gradient.
$$ \frac{\partial \log \mathrm{Pr}_{\theta}(\tau)}{\partial \theta} = \frac{\partial}{\partial \theta} \sum_{t=1}^{T} \underbrace{\log \pi_{\theta}(a_t|s_t)}_{\text{policy}} + \frac{\partial}{\partial \theta} \sum_{t=1}^{T} \underbrace{\log \Pr(s_{t+1}|s_t,a_t)}_{\text{dynamics}} $$
The policy component, $$\log \pi_{\theta}(a_t|s_t)$$, reflects the log-probability of choosing action $$a_t$$ in state $$s_t$$, parameterized by $$\theta$$. The dynamics component, $$\log \Pr(s_{t+1}|s_t,a_t)$$, captures the environment's transition probabilities to the next state $$s_{t+1}$$, given the current state and action.

Decomposition of the Trajectory Log-Probability Gradient

In policy gradient methods, a common objective function to maximize is formulated using the advantage function, $$A(s_t, a_t)$$, to improve training stability. This objective, denoted as $$U(\tau; \theta)$$, is expressed as the sum over a trajectory of the log-probabilities of actions multiplied by their corresponding advantage values: $$U(\tau; \theta) = \sum_{t=1}^{T} \log \pi_{\theta}(a_t|s_t)A(s_t, a_t)$$ Here: - $$\pi_{\theta}(a_t|s_t)$$ is the policy, which gives the probability of taking action $$a_t$$ in state $$s_t$$. - $$A(s_t, a_t)$$ is the advantage function, which measures how much better action $$a_t$$ is compared to the expected value in state $$s_t$$. Maximizing this objective via gradient ascent encourages the policy to take actions that have a higher-than-average expected return.

Policy Gradient Objective with Advantage Function

The general form of the policy gradient is an expectation over trajectories sampled from the policy. By treating every trajectory $$\tau$$ in a sampled dataset $$\mathcal{D}$$ (with size $$m$$ or $$|\mathcal{D}|$$) as equally probable, we use the practical estimator:
$$ \frac{\partial J(\theta)}{\partial \theta} = \frac{1}{m} \sum_{\tau \in \mathcal{D}} \frac{\partial \log \mathrm{Pr}_{\theta}(\tau)}{\partial \theta} R(\tau) $$
By decomposing the sequence probability and dropping the dynamics gradient (which does not depend on $$\theta$$), and knowing that the cumulative reward is $$R(\tau) = \sum_{t=1}^{T} r_t$$, this objective expands to:
$$ \frac{\partial J(\theta)}{\partial \theta} = \frac{1}{|\mathcal{D}|} \sum_{\tau \in \mathcal{D}} \frac{\partial}{\partial \theta} \Big( \sum_{t=1}^{T} \log \pi_{\theta}(a_t|s_t) \sum_{t=1}^{T} r_t \Big) $$
This formulation highlights that the reward function $$R(\tau)$$ does not need to be differentiable for optimization.

Policy Gradient Estimate under Uniform Trajectory Probability

The score function in policy gradient methods is the gradient of the log-probability of a trajectory $\tau$ with respect to the policy parameters $\theta$. It is expressed as $\nabla_{\theta} \log \text{Pr}_{\theta}(\tau)$ or, using partial derivative notation, $\frac{\partial \log \text{Pr}_{\theta}(\tau)}{\partial \theta}$. This term is fundamental to deriving the policy gradient objective, as it indicates the direction in the parameter space that increases the probability of a given trajectory.

Score Function in Policy Gradients

During the derivation of the policy performance gradient, a key step transforms the expression `Σ [∂Pr_θ(τ)/∂θ] R(τ)` into a form that includes the term `∂log Pr_θ(τ)/∂θ`. What is the primary analytical purpose of this transformation?

The following equations represent key steps in deriving the policy gradient. Arrange them in the correct logical order, starting from the initial gradient of the objective function to its final form as an expectation. Note: J(θ) is the objective function, Pr_θ(τ) is the probability of a trajectory τ under policy parameters θ, and R(τ) is the reward for that trajectory.

A student is attempting to derive the policy gradient objective function. Their derivation is shown below. Identify the specific mathematical error in their steps and explain why this error introduces a fundamental problem that the standard derivation avoids.

**Derivation Steps:**
1. Objective Function: $J(\theta) = \sum_{\tau} \text{Pr}_{\theta}(\tau)R(\tau)$
2. Gradient Calculation: $\frac{\partial J(\theta)}{\partial \theta} = \sum_{\tau} \left[ \frac{\partial \text{Pr}_{\theta}(\tau)}{\partial \theta} R(\tau) + \text{Pr}_{\theta}(\tau) \frac{\partial R(\tau)}{\partial \theta} \right]$
3. Conclusion: The student stops here, concluding that for this gradient to be useful, the reward function $R(\tau)$ must be differentiable with respect to the policy parameters $\theta$.

Learn Before

Related