Google

The action-value function, or Q-value function, denoted as $$Q(s,a)$$, estimates the expected return when an agent starts from a specific state $$s$$, immediately takes a particular action $$a$$, and then adheres to a given policy $$\pi$$ for all subsequent decisions. It is formally defined as the expectation over all possible future trajectories:

$$Q(s,a) = \mathbb{E} \Big[ \sum_{t=0}^{\infty} \gamma^{t} r_t \ \big | \ s_0 = s, a_0 = a, \pi \Big]$$

Here, $$s_0 = s$$ designates the starting state, and $$a_0 = a$$ specifies the initial action taken. The parameter $$\gamma$$ represents the discount factor applied to future rewards, and $$r_t$$ is the reward obtained at time step $$t$$.

Action-Value Function Formula

The advantage function, $$A(s_t, a_t)$$, quantifies the relative benefit of taking a specific action $$a_t$$ compared to the expected value of following the policy from state $$s_t$$ onward. It is formally defined as the difference between the action-value function, $$Q(s_t, a_t)$$, and the state-value function, $$V(s_t)$$: $$ A(s_t, a_t) = Q(s_t, a_t) - V(s_t) $$ A positive advantage value suggests the action is better than the expected policy value, while a negative value suggests it is worse. This measure is crucial in methods like A2C as it helps focus policy updates on actions likely to improve performance.

Advantage Function Definition

Based on the scenario provided, calculate the expected total discounted reward for both Action A and Action B. Which action is preferable according to these calculations?

Calculating Action-Values in a Simple Environment

Match each component of the action-value function formula, $Q(s, a) = \mathbb{E}[\sum_{t=0}^{\infty} \gamma^t r_t | s_0 = s, a_0 = a, \pi]$, with its correct description.

An agent is being trained in an environment where it must choose between two initial actions from the same starting position. Action A leads to a short sequence of steps resulting in a small, immediate reward. Action B leads to a much longer sequence of steps resulting in a large, delayed reward. According to the action-value function formula, which calculates the expected total discounted reward for taking an action in a state, how would decreasing the discount factor (γ) from a high value (e.g

Learn Before

Related