The TRPO policy in this paper is parameterized by a gated recurrent unit (GRU), a recurrent neural network architecture whose hidden state summarizes past observations. Because the environment is a partially-observable Markov decision process (each step only reveals the previous item's identity, its binary recall outcome, and the elapsed time), the GRU's hidden state lets the policy integrate this history over time when choosing the next item to schedule.

Google

The authors combine **trust region policy optimization (TRPO)** with a **gated recurrent unit (GRU)** to solve the partially-observable Markov decision process (POMDP) formulated for spaced-repetition scheduling. At each step, the policy's input is the identity of the previous item, a binary recall outcome, and the time elapsed since that item's last review; the output is the next item to schedule. To scale the policy to a much larger number of items, the authors apply a **random projection trick** to reduce the dimensionality of the item-identity input.

Algorithm (Accelerating Human Learning With Deep Reinforcement Learning)

TRPO is an optimization algorithm in reinforcement learning which uses gradient descent. TRPO builds an algorithm that is stable and guarantees monotonic improvement. "This algorithm is similar to natural policy gradient methods and is effective for optimizing large nonlinear policies such as neural networks"

TRPO has  better performance than the vanilla policy gradients as the length of the step size is easily defined. Additionally, it takes advantage of using old policies sampled distributions for optimizing new ones.



Trust Region Policy Optimization

**Random Projection** is the technique used for the dimensionality reduction.

Learn Before

Related