**EFC** (Exponential Forgetting Curve), **HLR** (Half-Life Regression) , **GPL** (Generalized Power Law)  models are implemented as OpenAI Gym. environments.

Environments
(Accelerating Human Learning With Deep Reinforcement Learning)

These are the parameters that were used by the authors:
1. n = 30 (number of items)
2. T = 200 (number of steps)
3. D = 5 (delay between steps (seconds))
4. For **EFL** (exponential forgetting curve)  sample item difficulty ($\theta$) is from the distribution:
$$log\theta \sim  N log(0.077, 1)$$

5. For **HLR** (Half-Life Regression) :
$$ \overrightarrow {\rm \theta} = (1, 1, 0, \theta_3 \sim N (0, 1)) $$
$ \overrightarrow {\rm x_i} = $  (num attempts, num correct, num incorrect, one-hot encoding of item i out of n items).

6. For **GPL** (Generalized Power-Law) 
$$a = \overrightarrow{\rm \alpha} = 0; d \sim N(1, 1);$$
$$ \log \overrightarrow {\rm d} \sim N(0,1 ), \log r \sim N (0, 0.01); W = 5;$$
$$  \theta_{2w} = \theta_{2w - 1} = \frac {1}{\sqrt W - w + 1} $$

7. For TRPO batch size is 4000, $\gamma = 0.99$; step_size = 0.01
8. For Recurrent Network Policy number of hidden layers is 32

Implementation Details
(Accelerating Human Learning With Deep Reinforcement Learning)

**Can described DRL scheduling algorithm help students achieve their educational objectives better than baseline methods, under various assumptions about student learning? **

Research Question
(Accelerating Human Learning With Deep Reinforcement Learning)

1. TRPO performs better than the other baselines used on **GPL**.
2. TRPO sometimes outperforms the other baselines on **EFC** & **HLR** and vise-versa.
3. TRPO performs worse that threshold policy (this was described as unsurprising as threshold has an access to the latent parameters).

The authors claimed that the fact that TRPO sometimes performs worse that **EFL** and **HLR** is interesting. Thy state that additional hyperparameters  and policies can increase TRPO performance.

Analysis
(Accelerating Human Learning With Deep Reinforcement Learning)

The authors compare TPRO to other four baseline schedulers:
1. Random Policy
2. Leitner System
3.  Variant SuperMemo
4.  Threshold-based Policy

As it was described in the research paper a threshold-based policy has direct access the student simulator's parameters when it calculates recall likelihoods, and it was used as an upper bound in the authors experiments.

Baselines
(Accelerating Human Learning With Deep Reinforcement Learning)

1. Research Question
2. Environment
3. Baselines
4. Implementation details
5. Analysis

San Diego State University

1. Introduction
2. Related Work
3. Background
4.  Spaced Repetition via Model-Free Reinforcement Learning
5. Experiments
6. Discussion

Accelerating Human Learning
with Deep Reinforcement Learning

http://people.eecs.berkeley.edu/~reddy/files/DRL_Tutor_NIPS17_MT_Workshop.pdf

Reference for Accelerating Human Learning with Deep Reinforcement Learning

The introduction explains the importance of increasing memorization in order to deepen knowledge. Therefore, there was done a lot of work in order to gain better understanding of how human memorization works. Some memory models tried to formalize so-called spacing effect, meaning that spaced review improves the memorization. 

Nowadays, learning tools like flashcards are very popular. But systems similar to SuperMemo lack reasoning or mathematical understanding of the above-mentioned problem.  Review scheduling can be quite challenging in the similar applications. This is related to the trade-off between showing a new content or the old ones.

As DRL achieved quite good results in training agent, the authors of this paper assume that similar can be achieved if they formulate spaced repetition as a task. The agent will focus on students performance and will try to understand the factors that influence student learning and use them in order to improve student learning. The authors aim to use review scheduling algorithm with model-free reinforcement learning where neural network tries to approximate following three teaching policies:

1. Directly operates on raw observations of outcomes and intervals between reviews
2. Scales to large numbers of items
3. Easily adapts to different learning objectives and student models.

The key problem will be DRL sample complexity. As the authors claim: "learning a policy requires a large number of interactions with the environment. Training an autonomous tutor from scratch using a large number of interactions with real students is infeasible."

Introduction 
(Accelerating Human Learning With Deep Reinforcement Learning)

There was a lot of previous work done in order to improve student learning. I will list here only some of the works that were provided in the research paper. The idea to use DRL to solve this problem is not brand-spark new. In 2014 several researchers developed "a reinforcement learning agent for an adaptive tutoring task in a challenging realworld educational game application, using an offline importance sampling-based method to select representations and hyperpameter settings without expensive online experiments on real students."

The Leithner system (the German science journal gets credit for it, this journal proposed efficient flashcard usage) "uses a network of first-in-first-out queues to coarsely prioritize items by novelty and difficulty." This system is quite old, it was developed in the 1970s.

Most of the previous works were closely related with machine teaching and knowledge tracing problems. The former one attempts to find optimal set for the 
provided algorithm and the latter one observes and measures how student's knowledge changes. 

Related Work
(Accelerating Human Learning With Deep Reinforcement Learning)

Spaced Repetition via Model-Free Reinforcement Learning
(Accelerating Human Learning With Deep Reinforcement Learning)

In order to improve the performance of the model, the authors state that it is crucial to run user studies with real students.  Also they think that if they would implement more intelligent initialization it would help the model a lot.
Additionally, the policy that would work on dynamic data (students often add new items which as a result decreases priority of the older ones) would make an advance.
 

Discussion
(Accelerating Human Learning With Deep Reinforcement Learning)

Experiments
(Accelerating Human Learning With Deep Reinforcement Learning)

Here the authors describe three previous probabilistic methods of the human memory:

**Exponential Forgetting Curve**  - one of the oldest human memory models.
The authors use parameterization where user either forgets information completely or retains it and the probability of the recall can be expressed with this function:  

"where θ ∈ R+ is the item difficulty, D ∈ R+ is the time elapsed since the item was last reviewed by the student, and S ∈ R+ is the student’s memory strength for the item." S is set to the amount of trials. 

                                       $$Z ~ Bernoulli(exp(-\theta \frac {D}{S}))$$
**Half-Life Regression** - here unlike **exponential forgetting curve** we don't have the parameter for the difficulty here, instead here we have ~x ∈ X which contains information about student's study history and model parameters θ ∈ Θ . This is how formula looks like:
                                 $$ S = exp(\overrightarrow { \rm \theta} \overrightarrow {\rm x})$$

In order to encode the number of attempts, correct/incorrect answers and the identity  the authors set $X = \N ^{3} \times {0, 1} ^{n}$. By dropping difficulty of the item we are not losing any information as the difficulty "is absorbed into the memory strength via the coefficients of the item identity indicator features."

**Generalized Power Law** (paper describes this part precisely, couldn't be broken down into smaller parts) here we have different likelihood for the recall:


On the last formula from the picture we have a and d  which are parameters for the student ability and the item difficulty.


Background
(Accelerating Human Learning With Deep Reinforcement Learning)

**Motivation**

1. Popularity of e-learning websites/applications
2. "Recommending content and personalizing the learning sessions is a highly sought after tasks on these platforms"
3. Explore the opportunities to substitute manually selected criteria

**Problem**

4. Deeper explore and extend work Accelerating Human Learning with Deep Reinforcement Learning by several modifications.



Motivation and Problem Description (Using deep reinforcement learning for personalizing review sessions on e-learning platforms with spaced repetition)

Learn Before

Related

Learn After