Google

A state-action sequence, or trajectory (τ), documents the path an agent takes through an environment. While the core concept is consistent, the notation used to represent these sequences can vary. For instance, a trajectory may be denoted as starting from time step 1, such as τ = {(s₁, a₁), (s₂, a₂), ...}, often to align with other notation in a specific context, like sequence prediction. Alternatively, it is common in reinforcement learning literature to see trajectories starting from time step 0, with varying lengths, such as τ = {(s₀, a₀), ..., (sT, aT)} or τ = {(s₀, a₀), ..., (sT-₁, aT-₁)}. These notational differences are a matter of convention and do not alter the fundamental principles or models being discussed.

Notational Variations in State-Action Sequences (Trajectories)

The cumulative reward, also referred to as the return, for a specific state-action sequence (or trajectory) is determined by adding together all the individual rewards accumulated during that sequence. If we denote a trajectory consisting of $$T$$ time steps as $$\tau = \{(s_1,a_1),...,(s_T,a_T)\}$$, the total cumulative reward $$R(\tau)$$ is formally calculated as the sum of the rewards $$r_t$$ from the first step to the final step:

$$R(\tau) = \sum_{t=1}^{T} r_t$$

Cumulative Reward of a Trajectory

An agent in an environment completes a sequence of two actions. It starts in an initial state `s₀`, performs action `a₀` to reach state `s₁`, and then performs action `a₁` to reach the final state `s₂`. Which of the following notations correctly represents the full sequence of state-action pairs, often called a trajectory (τ)?

An agent starts in an initial state, takes a single action, and arrives at a final state. Two different notations are proposed to represent this entire sequence of events:

Notation 1: τ = {(s₀, a₀)}
Notation 2: τ = {(s₀, a₀), (s₁, a₁)}

For each notation, explain one specific reason why it could be considered an incomplete or potentially misleading representation of the described sequence.

Critiquing Trajectory Notations

An agent interacts with an environment for a total of `T` time steps, resulting in a sequence of states and actions. Match each mathematical notation for this sequence (trajectory, τ) to the description that accurately characterizes its structure and length.

Learn Before

Related