http://people.eecs.berkeley.edu/~reddy/files/DRL_Tutor_NIPS17_MT_Workshop.pdf

San Diego State University

1. Introduction
2. Related Work
3. Background
4.  Spaced Repetition via Model-Free Reinforcement Learning
5. Experiments
6. Discussion

Accelerating Human Learning
with Deep Reinforcement Learning

Reference for Accelerating Human Learning with Deep Reinforcement Learning

The introduction explains the importance of increasing memorization in order to deepen knowledge. Therefore, there was done a lot of work in order to gain better understanding of how human memorization works. Some memory models tried to formalize so-called spacing effect, meaning that spaced review improves the memorization. 

Nowadays, learning tools like flashcards are very popular. But systems similar to SuperMemo lack reasoning or mathematical understanding of the above-mentioned problem.  Review scheduling can be quite challenging in the similar applications. This is related to the trade-off between showing a new content or the old ones.

As DRL achieved quite good results in training agent, the authors of this paper assume that similar can be achieved if they formulate spaced repetition as a task. The agent will focus on students performance and will try to understand the factors that influence student learning and use them in order to improve student learning. The authors aim to use review scheduling algorithm with model-free reinforcement learning where neural network tries to approximate following three teaching policies:

1. Directly operates on raw observations of outcomes and intervals between reviews
2. Scales to large numbers of items
3. Easily adapts to different learning objectives and student models.

The key problem will be DRL sample complexity. As the authors claim: "learning a policy requires a large number of interactions with the environment. Training an autonomous tutor from scratch using a large number of interactions with real students is infeasible."

Introduction 
(Accelerating Human Learning With Deep Reinforcement Learning)

There was a lot of previous work done in order to improve student learning. I will list here only some of the works that were provided in the research paper. The idea to use DRL to solve this problem is not brand-spark new. In 2014 several researchers developed "a reinforcement learning agent for an adaptive tutoring task in a challenging realworld educational game application, using an offline importance sampling-based method to select representations and hyperpameter settings without expensive online experiments on real students."

The Leithner system (the German science journal gets credit for it, this journal proposed efficient flashcard usage) "uses a network of first-in-first-out queues to coarsely prioritize items by novelty and difficulty." This system is quite old, it was developed in the 1970s.

Most of the previous works were closely related with machine teaching and knowledge tracing problems. The former one attempts to find optimal set for the 
provided algorithm and the latter one observes and measures how student's knowledge changes. 

Related Work
(Accelerating Human Learning With Deep Reinforcement Learning)

Spaced Repetition via Model-Free Reinforcement Learning
(Accelerating Human Learning With Deep Reinforcement Learning)

In order to improve the performance of the model, the authors state that it is crucial to run user studies with real students.  Also they think that if they would implement more intelligent initialization it would help the model a lot.
Additionally, the policy that would work on dynamic data (students often add new items which as a result decreases priority of the older ones) would make an advance.
 

Discussion
(Accelerating Human Learning With Deep Reinforcement Learning)

1. Research Question
2. Environment
3. Baselines
4. Implementation details
5. Analysis

Experiments
(Accelerating Human Learning With Deep Reinforcement Learning)

Here the authors describe three previous probabilistic methods of the human memory:

**Exponential Forgetting Curve**  - one of the oldest human memory models.
The authors use parameterization where user either forgets information completely or retains it and the probability of the recall can be expressed with this function:  

"where θ ∈ R+ is the item difficulty, D ∈ R+ is the time elapsed since the item was last reviewed by the student, and S ∈ R+ is the student’s memory strength for the item." S is set to the amount of trials. 

                                       $$Z ~ Bernoulli(exp(-\theta \frac {D}{S}))$$
**Half-Life Regression** - here unlike **exponential forgetting curve** we don't have the parameter for the difficulty here, instead here we have ~x ∈ X which contains information about student's study history and model parameters θ ∈ Θ . This is how formula looks like:
                                 $$ S = exp(\overrightarrow { \rm \theta} \overrightarrow {\rm x})$$

In order to encode the number of attempts, correct/incorrect answers and the identity  the authors set $X = \N ^{3} \times {0, 1} ^{n}$. By dropping difficulty of the item we are not losing any information as the difficulty "is absorbed into the memory strength via the coefficients of the item identity indicator features."

**Generalized Power Law** (paper describes this part precisely, couldn't be broken down into smaller parts) here we have different likelihood for the recall:


On the last formula from the picture we have a and d  which are parameters for the student ability and the item difficulty.


Background
(Accelerating Human Learning With Deep Reinforcement Learning)

**Motivation**

1. Popularity of e-learning websites/applications
2. "Recommending content and personalizing the learning sessions is a highly sought after tasks on these platforms"
3. Explore the opportunities to substitute manually selected criteria

**Problem**

4. Deeper explore and extend work Accelerating Human Learning with Deep Reinforcement Learning by several modifications.



Motivation and Problem Description (Using deep reinforcement learning for personalizing review sessions on e-learning platforms with spaced repetition)

Learn Before

Related