Equivalence of Surrogate and On-Policy Gradients at the Reference Point

In a reinforcement learning scenario, an agent is in a particular state and has two possible actions, Action A and Action B. The agent's current parameterized policy assigns a non-zero probability to both actions. After sampling several trajectories, the agent estimates that the expected cumulative reward for taking Action A from this state is +10, while the expected cumulative reward for taking Action B from this state is -5. Based on the fundamental principle of updating a policy to maximize expected returns, how will the gradient update affect the probabilities of these actions?

Diagnosing Learning Issues in Policy Gradients

An agent's learning process involves updating its decision-making parameters (θ) based on experience. The update rule is proportional to the expression: Σ_s ρ(s) Σ_a ∇_θ π(s,a) Q(s,a). Match each mathematical component from this expression to its conceptual role in guiding the learning update.