Advantage Function Definition

In a reinforcement learning algorithm, a baseline is subtracted from the total reward to stabilize the learning process. Consider two different baseline strategies:

Strategy 1: Use a single, fixed value for the baseline, such as the average total reward calculated over many past episodes. Strategy 2: Use a dynamic value for the baseline that is equal to the expected future reward from the agent's current state.

Why is Strategy 2 generally more effective at reducing the variance of the policy updates compared to Strategy 1?

Evaluating Actions with a State-Value Baseline

Analyzing the Impact of a State-Value Baseline

Advantage Function Definition

An agent is being trained in an environment where it must choose between two initial actions from the same starting position. Action A leads to a short sequence of steps resulting in a small, immediate reward. Action B leads to a much longer sequence of steps resulting in a large, delayed reward. According to the action-value function formula, which calculates the expected total discounted reward for taking an action in a state, how would decreasing the discount factor (γ) from a high value (e.g., 0.99) to a very low value (e.g., 0.1) most likely influence the agent's learned behavior?

Calculating Action-Values in a Simple Environment

Match each component of the action-value function formula, $Q(s, a) = \mathbb{E}[\sum_{t=0}^{\infty} \gamma^t r_t | s_0 = s, a_0 = a, \pi]$, with its correct description.