Policy Gradient Estimation from Sampled Trajectories

An agent is being trained using a policy gradient method. The theoretical objective gradient is expressed as an expectation over trajectories τ sampled from the policy π_θ:

∇J(θ) = E_{τ~π_θ}[ (∇_θ log Pr_θ(τ)) R(τ) ]

In practice, this is estimated from a batch of |D| sampled trajectories using the following formula:

∇J(θ) ≈ (1/|D|) Σ_{τ∈D} (∇_θ log Pr_θ(τ)) R(τ)

What key assumption allows for the transition from the theoretical expectation to this practical sample mean estimator?

Policy Gradient with Baseline

Reward-to-Go

An agent is being trained using a policy gradient method. A batch of data D is collected, containing exactly two trajectories, τ_1 and τ_2.

• Trajectory τ_1 has a total reward R(τ_1) = 10.
• Trajectory τ_2 has a total reward R(τ_2) = -5.

The gradient of the log-probability for each trajectory with respect to the policy parameters θ is denoted as ∇_θ log Pr_θ(τ_1) and ∇_θ log Pr_θ(τ_2), respectively.

Based on the standard practical estimator for the policy gradient, which of the following expressions correctly represents the estimated gradient ∇J(θ) for this batch?

Policy Gradient Estimate with Baseline

Baseline's Role in Centering Rewards and Reducing Gradient Variance

State-Value Function as a Baseline

Baseline's Impact on Reward Variance vs. Gradient Estimate Variance

An engineer is training two reinforcement learning agents (Agent A and Agent B) on the same task using a policy gradient method. The environment has a wide range of possible total rewards, from highly negative to highly positive. Agent A's learning algorithm directly uses the total reward received after each episode to update its policy. Agent B's algorithm first subtracts a constant value (equal to the average total reward observed so far) from the total reward before using it for the update. What is the most likely difference in the training process between Agent A and Agent B?

Benefit of a Baseline in a Positive-Reward Environment

A reinforcement learning agent is being trained in a specialized environment where the total reward for any complete episode consistently falls within a narrow range of 95 to 105. The training algorithm uses a policy gradient method and incorporates a baseline by subtracting the long-term average reward (approximately 100) from each episode's total reward before performing an update. Which statement best evaluates the utility of this baseline in this specific scenario?

Policy Gradient with Baseline