Evaluating an Action's Performance

An agent in a given state s takes an action a. The sequence of rewards it receives from that point until the end of the episode sums to a total of 10. The pre-calculated value for state s, representing the average expected sum of future rewards from that state, is 15. Based on this information, what can be concluded about the action a?

Comparing Action Quality

An agent is in a state 'S' and must choose between two policies, Policy A and Policy B. The sequence of rewards the agent will receive after starting in state 'S' and following each policy is deterministic and known:

• Policy A Reward Sequence: [+10, +1, +1, +1, ...]
• Policy B Reward Sequence: [+3, +3, +3, +3, ...]

Given the formula for the value of a state, $V(s) = \mathbb{E}[\sum_{t=0}^{\infty} \gamma^t r_t | s_0 = s, \pi]$, which of the following statements correctly analyz

Calculating State Value in a Deterministic Environment

Temporal Difference (TD) Error as an Advantage Function Estimator

An agent is in a state 'S' and follows a fixed policy. From this state, the environment is stochastic: there is a 50% chance the agent will enter a trajectory with a reward sequence of [+10, 0, 0, ...] and a 50% chance it will enter a different trajectory with a reward sequence of [0, +10, 0, ...]. Given the state-value formula $V(s) = \mathbb{E}[\sum_{t=0}^{\infty} \gamma^t r_t | s_0 = s, \pi]$ and a discount factor (γ) of 0.9, what is the value of state 'S'?

Value-Based Reward Shaping Formula