In the context of optimizing spaced repetition via model-free reinforcement learning, the environment is formulated as a partially-observable Markov decision process.

* **State Space ($$S$$)**: Depends on the student model.
  * For the **EFC** (exponential forgetting curve) model: $$S = \mathbb{R}_{+}^{3n}$$, encoding item difficulty, delay, and memory strength.
  * For the **HLR** (half-life regression) model: $$S = \theta \times (\mathbb{R}_{+} \times X)^n$$, encoding model parameters, delay, and memory strength.
  * For the **GPL** (generalized-power-law) model: $$S = \mathbb{R} \times (\mathbb{R} \times \mathbb{N}^{2W})^n$$, encoding student ability, item difficulty, number of attempts, and number of correct answers over $$W$$ windows for $$n$$ items.
* **Observation Space**: The agent can only access observations, not the full state. At every step, the observation set stores whether the student remembered the shown item or not: $$O(z | s, \alpha) = P [Z_{\alpha} = z | s]$$
* **Agent/Action Space**: Consists of $$n$$ items that can be shown to the student.
* **Reward Function ($$\mathcal{R}$$)**: Depending on the goal, there are two distinct functions:
  * Maximizing expected items recalled: $$\mathcal{R}(s, \bullet) = \sum_{i=1}^n P [Z_i = 1 | s]$$
  * Maximizing the likelihood of recalling all items: $$\mathcal{R}(s, \bullet) = \sum_{i=1}^n \log P [Z_i = 1 | s]$$

Note: The discount factor ($$\gamma$$) influences agent actions. A smaller $$\gamma$$ encourages intensive studying, while a larger $$\gamma$$ focuses on long-lasting learning.

Formulation (Accelerating Human Learning with Deep Reinforcement Learning)

The authors combine **trust region policy optimization (TRPO)** with a **gated recurrent unit (GRU)** to solve the partially-observable Markov decision process (POMDP) formulated for spaced-repetition scheduling. At each step, the policy's input is the identity of the previous item, a binary recall outcome, and the time elapsed since that item's last review; the output is the next item to schedule. To scale the policy to a much larger number of items, the authors apply a **random projection trick** to reduce the dimensionality of the item-identity input.

Algorithm (Accelerating Human Learning With Deep Reinforcement Learning)

San Diego State University

1. Introduction
2. Related Work
3. Background
4.  Spaced Repetition via Model-Free Reinforcement Learning
5. Experiments
6. Discussion

Accelerating Human Learning
with Deep Reinforcement Learning

http://people.eecs.berkeley.edu/~reddy/files/DRL_Tutor_NIPS17_MT_Workshop.pdf

Reference for Accelerating Human Learning with Deep Reinforcement Learning

The introduction explains the importance of increasing memorization in order to deepen knowledge. Therefore, there was done a lot of work in order to gain better understanding of how human memorization works. Some memory models tried to formalize so-called spacing effect, meaning that spaced review improves the memorization. 

Nowadays, learning tools like flashcards are very popular. But systems similar to SuperMemo lack reasoning or mathematical understanding of the above-mentioned problem.  Review scheduling can be quite challenging in the similar applications. This is related to the trade-off between showing a new content or the old ones.

As DRL achieved quite good results in training agent, the authors of this paper assume that similar can be achieved if they formulate spaced repetition as a task. The agent will focus on students performance and will try to understand the factors that influence student learning and use them in order to improve student learning. The authors aim to use review scheduling algorithm with model-free reinforcement learning where neural network tries to approximate following three teaching policies:

1. Directly operates on raw observations of outcomes and intervals between reviews
2. Scales to large numbers of items
3. Easily adapts to different learning objectives and student models.

The key problem will be DRL sample complexity. As the authors claim: "learning a policy requires a large number of interactions with the environment. Training an autonomous tutor from scratch using a large number of interactions with real students is infeasible."

Introduction 
(Accelerating Human Learning With Deep Reinforcement Learning)

There was a lot of previous work done in order to improve student learning. I will list here only some of the works that were provided in the research paper. The idea to use DRL to solve this problem is not brand-spark new. In 2014 several researchers developed "a reinforcement learning agent for an adaptive tutoring task in a challenging realworld educational game application, using an offline importance sampling-based method to select representations and hyperpameter settings without expensive online experiments on real students."

The Leithner system (the German science journal gets credit for it, this journal proposed efficient flashcard usage) "uses a network of first-in-first-out queues to coarsely prioritize items by novelty and difficulty." This system is quite old, it was developed in the 1970s.

Most of the previous works were closely related with machine teaching and knowledge tracing problems. The former one attempts to find optimal set for the 
provided algorithm and the latter one observes and measures how student's knowledge changes. 

Related Work
(Accelerating Human Learning With Deep Reinforcement Learning)

1. Research Question
2. Environment
3. Baselines
4. Implementation details
5. Analysis

Experiments
(Accelerating Human Learning With Deep Reinforcement Learning)

Here the authors describe three previous probabilistic methods of the human memory:

**Exponential Forgetting Curve**  - one of the oldest human memory models.
The authors use parameterization where user either forgets information completely or retains it and the probability of the recall can be expressed with this function:  

"where θ ∈ R+ is the item difficulty, D ∈ R+ is the time elapsed since the item was last reviewed by the student, and S ∈ R+ is the student’s memory strength for the item." S is set to the amount of trials. 

                                       $$Z ~ Bernoulli(exp(-\theta \frac {D}{S}))$$
**Half-Life Regression** - here unlike **exponential forgetting curve** we don't have the parameter for the difficulty here, instead here we have ~x ∈ X which contains information about student's study history and model parameters θ ∈ Θ . This is how formula looks like:
                                 $$ S = exp(\overrightarrow { \rm \theta} \overrightarrow {\rm x})$$

In order to encode the number of attempts, correct/incorrect answers and the identity  the authors set $X = \N ^{3} \times {0, 1} ^{n}$. By dropping difficulty of the item we are not losing any information as the difficulty "is absorbed into the memory strength via the coefficients of the item identity indicator features."

**Generalized Power Law** (paper describes this part precisely, couldn't be broken down into smaller parts) here we have different likelihood for the recall:


On the last formula from the picture we have a and d  which are parameters for the student ability and the item difficulty.


Background
(Accelerating Human Learning With Deep Reinforcement Learning)

**Motivation**

1. Popularity of e-learning websites/applications
2. "Recommending content and personalizing the learning sessions is a highly sought after tasks on these platforms"
3. Explore the opportunities to substitute manually selected criteria

**Problem**

4. Deeper explore and extend work Accelerating Human Learning with Deep Reinforcement Learning by several modifications.



Motivation and Problem Description (Using deep reinforcement learning for personalizing review sessions on e-learning platforms with spaced repetition)

To improve model performance, the authors of "Accelerating Human Learning with Deep Reinforcement Learning" propose three future research directions:

1. **User Studies**: Running user studies with real students to evaluate actual learning outcomes and model behavior.
2. **Intelligent Initialization**: Implementing more sophisticated model initialization to accelerate learning and enhance early session recommendations.
3. **Dynamic Data Policies**: Designing policies capable of adapting to dynamic data streams where newly added items naturally shift the priority of older items in the student's review queue.

Learn Before

Related

Learn After