TRPO is an optimization algorithm in reinforcement learning which uses gradient descent. TRPO builds an algorithm that is stable and guarantees monotonic improvement. "This algorithm is similar to natural policy gradient methods and is effective for optimizing large nonlinear policies such as neural networks"

TRPO has  better performance than the vanilla policy gradients as the length of the step size is easily defined. Additionally, it takes advantage of using old policies sampled distributions for optimizing new ones.



Trust Region Policy Optimization

**Random Projection** is the technique used for the dimensionality reduction.

Random Projection

The authors use **trust region policy optimization** with gated recurrent unit in order to solve partially-observable Markov decision processes. As an input policy has the identity of the previous item, binary outcome(for recall), the time elapsed.   Then it gives an output for the next item. In order to scale to much larger amount of items, the authors used **random projection trick**.

San Diego State University

Spaced Repetition via Model-Free Reinforcement Learning
(Accelerating Human Learning With Deep Reinforcement Learning)

S is a state space which depends on the student model.

For **EFC** (exponential forgetting curve) state space $S = \R_{+}^{3n}$  where the item difficulty, delay and memory strength are encoded.  

For **HLR**(half-life regression)  state space $S = \theta \times (\R_{+} \times X)^n$ where model parameters, delay and memory strength are encoded.

For **GPL** (generalized-power-law)  state space $S = \R \times (\R \times \N^{2W})^n$ the student ability, item difficulty, number of attempts over W windows, and number of correct answers over W windows for n items are encoded.

It has to be denoted that agent can access observations and not the state.

For the agent we have agent space consisting of n items that can be shown to students. 
Transitions functions for the three models aren't directly provided.
Depending on the goal we have two distinctive rewarding functions.
For maximizing expected number of items recalled:
$$\R(s, \bullet) = \sum_{i=1}^n P [Z_i = 1 | s] $$
For maximizing the likelihood of recalling all items:

$$\R(s, \bullet) = \sum_{i=1}^n \log P [Z_i = 1 | s] $$

Depending on the discount factor $\gamma$  we will have different agent actions. For smaller one agent tries to help student to study intensively and for larger one it focuses on long-lasting learning. At every step, observation set stores information about whether student remembered show item or not.  For observation distribution we have following formula:
$$O(z | s, \alpha) = P [Z_{\alpha} = z | s] $$

Learn Before

Related

Learn After