A good way to understand reinforcement learning would be to think about how autonomous vehicles work. Instead of programming all of the rules it should follow (such as road signs, pedestrians, right of way, etc), it is much more easier and efficient to implement a reward/penalty system where the car learns all of the rules that it is supposed to abide by. Furthermore, it would be unreasonable to program individual rules because accidents can happen in unpredictable ways and rules/regulations are always changing and varies in different parts of the world. So, it is much better for autonomous cars to be always learning better ways to make decisions.

Reinforcement Learning Example - Autonomous Vehicles

Video games often involve completing levels, with increasing difficulty as you go through the game. In each level, you go in completely blind and make decisions that will either lead to you passing that level or failing. If you fail, you will now understand more about that particular level that will influence your decisions when you retry. In other words, you will understand what not to do, and try another approach. If you pass, you will understand what to do in future levels. This process is repeated until success. This is very similar to how reinforcement learning works. Rewards are given for successes, indicating that the decision(s) made are correct and should be learned for future use. Penalties are given for a failure, indicating that the decision(s) made should not be retried and instead a new approach should be used.

Reinforcement Learning Analogy - Video Games

Theobald, O. (2017). Machine learning for absolute beginners: a plain English introduction (2nd ed.). Scatterplot Press.

Online pdf: http://bmansoori.ir/book/Machine%20Learning%20For%20Absolute%20Beginners.pdf

Machine Learning for Absolute Beginners

Deep learning analyses a training set, identifies complex patterns and applies them to new data. 

Reinforcement learning works sequentially in an unknown environment━taking an action, evaluating the rewards, and adjusting the following actions accordingly.

Deep learning and reinforcement learning complement each other:

 - Reinforcement learning algorithms manage the sequential process of taking an action, evaluating the result, and selecting the next best action. However, they need a good mechanism to select the best action based on previous interactions.
 - Deep learning can be that mechanism━it is the most powerful method available today to learn the best outcome based on previous data.

 

Deep Learning vs. Reinforcement Learning

The fundamental concepts that form the basis of reinforcement learning are:

 - Agent
 - Environment
 - Action
 - State
 - Reward

The agent learns through repeated interaction with the environment. To be successful, the agent needs to:
 - Learn the interaction between states, actions, and subsequent rewards.
 - Determine which action will provide the optimal outcome.

Fundamental Concepts for Reinforcement Learning

(Encourage to continuously enrich and edit this node for sharing papers and their proposed useful ideas.)
1.Learning Action Representations for Reinforcement Learning Yash Chandak, Georgios Theocharous, James E. Kostas, Scott M. Jordan, Philip S. Thomas 

Reinforcement Learning Refrence and Cutting-edge Ideas

A team is developing a program to play a complex board game against human opponents. The program has no pre-existing data of past games to learn from. Instead, it is designed to learn by playing against itself repeatedly. After each game, the program receives a positive signal if it wins and a negative signal if it loses. Over time, it is expected to discover winning strategies on its own. Which of the following statements best analyzes why this learning approach is suitable for this task?

This scenario describes a classic reinforcement learning problem. Identify the Agent, Environment, State, Action, and Reward within this context. Then, explain how the reward structure incentivizes the agent to achieve the desired outcome of finding the exit quickly.

Robot Maze Navigation Strategy

A company is developing a system to recommend personalized news articles. The system's goal is to continuously adapt to a user's evolving interests. Two strategies are proposed:

**Strategy A:** Collect a massive, one-time dataset of user profiles and the articles they clicked on in the past. Train a model on this static dataset to predict which articles a new user is likely to click.

**Strategy B:** Design a system that learns through direct interaction. For each article it recommends, it receives a positive signal if the user clicks it and a negative signal if they ignore it. The system's objective is to maximize the positive signals it receives over time.

Evaluate these two strategies. Which one is better suited for creating a system that adapts to a user's *evolving* interests? Justify your evaluation by comparing the core mechanics of each strategy and their implications for long-term personalization.

Evaluating Learning Strategies for a Recommendation System

In reinforcement learning, the machine goes through trial and error processes where it is rewarded/penalized for the actions it performs. The machine's goal is to maximize the total reward by leveraging previous attempts to make the next decision. The model starts from completely random trials and eventually leads to something with very sophisticated tactics and skills. Compared to supervised/unsupervised learning, reinforcement learning is much more advanced. A reinforcement model will continuously learn, unlike supervised/unsupervised models which all have an endpoint after the training and test data phases.

University of Michigan - Ann Arbor

Google

San Diego State University

There are several categories for ML algorithms, including:
 - Supervised Learning
 - Semi-supervised Learning
 - Unsupervised Learning
 - Reinforcement Learning
 - Feature Learning (Representation Learning)

The types of machine learning algorithms differ in their approach, the type of data they input and output, and the type of task or problem that they are intended to solve.


Machine Learning Categories

1. Reinforcement Learning
2. Intelligent Tutoring Systems

History (Using deep reinforcement learning for personalizing review sessions on e-learning platforms with spaced repetition)

1. Models of Human Memory
2. Spaced Repetition
3. Leitner System
4. Supermemo System
5. Intelligent Tutoring System
6. Relation between Tutoring Systems and Student Learning
7. Reinforcement Learning
8. Trust Region Policy Optimization
9. Truncated Natural Policy Gradient
10. Recurrent Neural Networks

Related Theory (Using deep reinforcement learning for personalizing review sessions on e-learning platforms with spaced repetition)

Reference of Foundations of Large Language Models Course

Unsupervised machine learning methods find structure in unlabeled data.
They are used when we DON'T have prior knowledge about the categories.

Unsupervised statistical learning

Reinforcement Learning

- Feature learning, also called representation learning use machine learning to discover not only the mapping from representation to output but also the representation itself. A feature learning algorithm can discover a good set of features for a simple task in minutes, or for a complex task in hours to months. Feature learning can be either supervised or unsupervised.

- Representation of information to be processed has a great importance in day to day life since it eases the task of information processing. For example usage of Arabic numerals instead of Roman numerals for mathematical operations.

- In the context of machine learning an acceptable representation of data makes the future learning tasks easy. The successive learning tasks decide the choice of information representation.



Feature Learning (Representation Learning)

https://www.amazon.com/Master-Algorithm-Ultimate-Learning-Machine-ebook/dp/B012271YB2

The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World

•	Inductive reasoning 
•	Connectionism
•	Evolutionary Computation 
•	Bayes’ theorem 
•	Analogical modeling 

Key takeaway: Pedro Domingos believes the “Master algorithm” will be a mixture of different schools of machine learning, as each school of thought has strengths and weaknesses that challenge their ability to be applied universally. 


Machine learning schools of thought (as explained in ”The Master Algorithm” by Pedro Domingos):

What are the categories of machine learning algorithms?

Supervised learning is a type of machine learning problem where the objective is to predict a designated unknown label based on known inputs. This approach requires a training dataset consisting of examples for which the ground truth labels are already known, allowing the model to learn the mapping from inputs to classifications.

Supervised Learning

In the 21st century MOOC (Massive Open Online Courses) have proliferated like Coursera, Udemy, Edx and several applications like Duolingo, Quizzlet became quite popular.  The researchers have a keen interest to acquire this systems with more intelligence, so that learning process could be overseen. 

As it was described in the paper Computer-Assisted Instruction can increase students performance by 0.3 std while human tutors by 2. 

Intelligent Tutoring Systems combine AI, cognitive science and educational theory which makes researches in this field arduous.  

In the earlier studies 3 main modules have been defined in ITS:

1. **The expert Knowledge module** - this module can be thought as a source of the knowledge. The purpose of this module is to generate questions, answers and the solutions for the particular problem.

2. **The student model module**  - the student skills and knowledge which varies constantly. It is very important for the **tutor module** to know about students current level and skills.

3. **The tutor module** - the teaching strategy. This module has to decide which lesson to present to the student.

Recently, fourth module was added **The user interface module**:



Intelligent Tutoring Systems (Using deep reinforcement learning for personalizing review sessions on e-learning platforms with spaced repetition)

Here the authors describe three previous probabilistic methods of the human memory:

**Exponential Forgetting Curve**  - one of the oldest human memory models.
The authors use parameterization where user either forgets information completely or retains it and the probability of the recall can be expressed with this function:  

"where θ ∈ R+ is the item difficulty, D ∈ R+ is the time elapsed since the item was last reviewed by the student, and S ∈ R+ is the student’s memory strength for the item." S is set to the amount of trials. 

                                       $$Z ~ Bernoulli(exp(-\theta \frac {D}{S}))$$
**Half-Life Regression** - here unlike **exponential forgetting curve** we don't have the parameter for the difficulty here, instead here we have ~x ∈ X which contains information about student's study history and model parameters θ ∈ Θ . This is how formula looks like:
                                 $$ S = exp(\overrightarrow { \rm \theta} \overrightarrow {\rm x})$$

In order to encode the number of attempts, correct/incorrect answers and the identity  the authors set $X = \N ^{3} \times {0, 1} ^{n}$. By dropping difficulty of the item we are not losing any information as the difficulty "is absorbed into the memory strength via the coefficients of the item identity indicator features."

**Generalized Power Law** (paper describes this part precisely, couldn't be broken down into smaller parts) here we have different likelihood for the recall:


On the last formula from the picture we have a and d  which are parameters for the student ability and the item difficulty.


Background
(Accelerating Human Learning With Deep Reinforcement Learning)

Reviewing or practicing over time can have better results than reviewing in short periods of time.

Spaced Repetition

The method used in flashcards for the spaced repetition. 

**The Goal** focus more on those items with which students have more difficulties to recall and spend less time on the ones that student recalls.

Leitner System

This is the algorithm for the supermemo (**SP2**) which was described in the paper:

1. Split knowledge

2. E-Factor = 2.5, for all items

3. Repeat items using the following intervals:
    I(1):=1
    I(2):=6
    for n>2: I(n):=I(n-1)*EF
    where:
    I(n) - inter-repetition interval after the n-th repetition (in days),
    EF - E-Factor (easiness of memorizing and retaining a given item in memory)

4. There is grade scale 0-5:
     5 - PERFECT
     4 - Correct + hesitation
     3 - Correct + took more time
     2 - Incorrect + correct easy to recall
     1 - Incorrect + correct  remembered
     0 - Blackout

5. After this we change E-Factors using the formula:
     EF_new:=EF+(0.1-(5-q)*(0.08+(5-q)*0.02))
     q - 0-5 range; EF_new - new E-Factor, if EF < 1.3 $\rightarrow$ EF = 1.3

6. If q < 3 then repeat 1-5 without changing EF.

7. After on a specific day the session is finished, repeat the procedure until at least q=4 is achieved for all the items.

Supermemo System

It is extremely important for the tutoring systems to realize the student's current knowledge level and predict student's ability to answer/solve a particular task or question correctly and modify student's knowledge level accordingly. The tutoring system has to take care of providing questions. System should take lots of factors into account like: 

1. Duration (to learn material)
2. Manner (to learn material)

Relation between Tutoring Systems and Student learning

TRPO is an optimization algorithm in reinforcement learning which uses gradient descent. TRPO builds an algorithm that is stable and guarantees monotonic improvement. "This algorithm is similar to natural policy gradient methods and is effective for optimizing large nonlinear policies such as neural networks"

TRPO has  better performance than the vanilla policy gradients as the length of the step size is easily defined. Additionally, it takes advantage of using old policies sampled distributions for optimizing new ones.



Trust Region Policy Optimization

**Truncated Natural Policy Gradient** (TNPG) is the algorithm for computing gradient direction. It only needs to calculate $I(\theta)v$ where:

1. $I (\theta)$ is Fisher Information Matrix
2. v is arbitrary vector.

TNPG improves natural policy gradient but it has a high computational cost. Usually, they are efficient when the space of the parameters is high dimensional. 

Truncated Natural Policy Gradient

Recurrent neural networks, or RNNs, are a family of neural networks for processing sequential data. A recurrent neural network is specialized for processing a sequence of inputs $x^{(1)}, . . . , x^{(τ)}$ and each time add additional layers of comprehension on top of the previous inputs. 


Learn Before

Related

Learn After