S is a state space which depends on the student model.

For EFC (exponential forgetting curve) state space $S = \R_{+}^{3n}$ where the item difficulty, delay and memory strength are encoded.

For HLR(half-life regression) state space $S = \theta \times (\R_{+} \times X)^n$ where model parameters, delay and memory strength are encoded.

For GPL (generalized-power-law) state space $S = \R \times (\R \times \N^{2W})^n$ the student ability, item difficulty, number of attempts over W windows, and number of correct answers over W windows for n items are encoded.

It has to be denoted that agent can access observations and not the state.

For the agent we have agent space consisting of n items that can be shown to students. Transitions functions for the three models aren't directly provided. Depending on the goal we have two distinctive rewarding functions. For maximizing expected number of items recalled: $\R(s, \bullet) = \sum_{i=1}^n P [Z_i = 1 | s]$ For maximizing the likelihood of recalling all items:

$\R(s, \bullet) = \sum_{i=1}^n \log P [Z_i = 1 | s]$

Depending on the discount factor $\gamma$ we will have different agent actions. For smaller one agent tries to help student to study intensively and for larger one it focuses on long-lasting learning. At every step, observation set stores information about whether student remembered show item or not. For observation distribution we have following formula: $O(z | s, \alpha) = P [Z_{\alpha} = z | s]$