S is a state space which depends on the student model.

For **EFC** (exponential forgetting curve) state space $S = \R_{+}^{3n}$  where the item difficulty, delay and memory strength are encoded.  

For **HLR**(half-life regression)  state space $S = \theta \times (\R_{+} \times X)^n$ where model parameters, delay and memory strength are encoded.

For **GPL** (generalized-power-law)  state space $S = \R \times (\R \times \N^{2W})^n$ the student ability, item difficulty, number of attempts over W windows, and number of correct answers over W windows for n items are encoded.

It has to be denoted that agent can access observations and not the state.

For the agent we have agent space consisting of n items that can be shown to students. 
Transitions functions for the three models aren't directly provided.
Depending on the goal we have two distinctive rewarding functions.
For maximizing expected number of items recalled:
$$\R(s, \bullet) = \sum_{i=1}^n P [Z_i = 1 | s] $$
For maximizing the likelihood of recalling all items:

$$\R(s, \bullet) = \sum_{i=1}^n \log P [Z_i = 1 | s] $$

Depending on the discount factor $\gamma$  we will have different agent actions. For smaller one agent tries to help student to study intensively and for larger one it focuses on long-lasting learning. At every step, observation set stores information about whether student remembered show item or not.  For observation distribution we have following formula:
$$O(z | s, \alpha) = P [Z_{\alpha} = z | s] $$

Formulation 
(Accelerating Human Learning With Deep Reinforcement Learning)

The authors use **trust region policy optimization** with gated recurrent unit in order to solve partially-observable Markov decision processes. As an input policy has the identity of the previous item, binary outcome(for recall), the time elapsed.   Then it gives an output for the next item. In order to scale to much larger amount of items, the authors used **random projection trick**.

Algorithm
(Accelerating Human Learning With Deep Reinforcement Learning)

San Diego State University

1. Introduction
2. Related Work
3. Background
4.  Spaced Repetition via Model-Free Reinforcement Learning
5. Experiments
6. Discussion

Accelerating Human Learning
with Deep Reinforcement Learning

http://people.eecs.berkeley.edu/~reddy/files/DRL_Tutor_NIPS17_MT_Workshop.pdf

Reference for Accelerating Human Learning with Deep Reinforcement Learning

The introduction explains the importance of increasing memorization in order to deepen knowledge. Therefore, there was done a lot of work in order to gain better understanding of how human memorization works. Some memory models tried to formalize so-called spacing effect, meaning that spaced review improves the memorization. 

Nowadays, learning tools like flashcards are very popular. But systems similar to SuperMemo lack reasoning or mathematical understanding of the above-mentioned problem.  Review scheduling can be quite challenging in the similar applications. This is related to the trade-off between showing a new content or the old ones.

As DRL achieved quite good results in training agent, the authors of this paper assume that similar can be achieved if they formulate spaced repetition as a task. The agent will focus on students performance and will try to understand the factors that influence student learning and use them in order to improve student learning. The authors aim to use review scheduling algorithm with model-free reinforcement learning where neural network tries to approximate following three teaching policies:

1. Directly operates on raw observations of outcomes and intervals between reviews
2. Scales to large numbers of items
3. Easily adapts to different learning objectives and student models.

The key problem will be DRL sample complexity. As the authors claim: "learning a policy requires a large number of interactions with the environment. Training an autonomous tutor from scratch using a large number of interactions with real students is infeasible."

Introduction 
(Accelerating Human Learning With Deep Reinforcement Learning)

There was a lot of previous work done in order to improve student learning. I will list here only some of the works that were provided in the research paper. The idea to use DRL to solve this problem is not brand-spark new. In 2014 several researchers developed "a reinforcement learning agent for an adaptive tutoring task in a challenging realworld educational game application, using an offline importance sampling-based method to select representations and hyperpameter settings without expensive online experiments on real students."

The Leithner system (the German science journal gets credit for it, this journal proposed efficient flashcard usage) "uses a network of first-in-first-out queues to coarsely prioritize items by novelty and difficulty." This system is quite old, it was developed in the 1970s.

Most of the previous works were closely related with machine teaching and knowledge tracing problems. The former one attempts to find optimal set for the 
provided algorithm and the latter one observes and measures how student's knowledge changes. 

Related Work
(Accelerating Human Learning With Deep Reinforcement Learning)

Spaced Repetition via Model-Free Reinforcement Learning
(Accelerating Human Learning With Deep Reinforcement Learning)

In order to improve the performance of the model, the authors state that it is crucial to run user studies with real students.  Also they think that if they would implement more intelligent initialization it would help the model a lot.
Additionally, the policy that would work on dynamic data (students often add new items which as a result decreases priority of the older ones) would make an advance.
 

Discussion
(Accelerating Human Learning With Deep Reinforcement Learning)

1. Research Question
2. Environment
3. Baselines
4. Implementation details
5. Analysis

Experiments
(Accelerating Human Learning With Deep Reinforcement Learning)

Here the authors describe three previous probabilistic methods of the human memory:

**Exponential Forgetting Curve**  - one of the oldest human memory models.
The authors use parameterization where user either forgets information completely or retains it and the probability of the recall can be expressed with this function:  

"where θ ∈ R+ is the item difficulty, D ∈ R+ is the time elapsed since the item was last reviewed by the student, and S ∈ R+ is the student’s memory strength for the item." S is set to the amount of trials. 

                                       $$Z ~ Bernoulli(exp(-\theta \frac {D}{S}))$$
**Half-Life Regression** - here unlike **exponential forgetting curve** we don't have the parameter for the difficulty here, instead here we have ~x ∈ X which contains information about student's study history and model parameters θ ∈ Θ . This is how formula looks like:
                                 $$ S = exp(\overrightarrow { \rm \theta} \overrightarrow {\rm x})$$

In order to encode the number of attempts, correct/incorrect answers and the identity  the authors set $X = \N ^{3} \times {0, 1} ^{n}$. By dropping difficulty of the item we are not losing any information as the difficulty "is absorbed into the memory strength via the coefficients of the item identity indicator features."

**Generalized Power Law** (paper describes this part precisely, couldn't be broken down into smaller parts) here we have different likelihood for the recall:


On the last formula from the picture we have a and d  which are parameters for the student ability and the item difficulty.


Background
(Accelerating Human Learning With Deep Reinforcement Learning)

**Motivation**

1. Popularity of e-learning websites/applications
2. "Recommending content and personalizing the learning sessions is a highly sought after tasks on these platforms"
3. Explore the opportunities to substitute manually selected criteria

**Problem**

4. Deeper explore and extend work Accelerating Human Learning with Deep Reinforcement Learning by several modifications.



Motivation and Problem Description (Using deep reinforcement learning for personalizing review sessions on e-learning platforms with spaced repetition)

Learn Before

Related

Learn After