https://cims.nyu.edu/~donev/Teaching/WrittenOral/Projects/XintianHan-WrittenAndOral.pdf
https://www.davidsilver.uk/teaching/     
(David Silver's tutorial of reinforcement learning)
http://rail.eecs.berkeley.edu/deeprlcourse/
https://katefvision.github.io/
http://incompleteideas.net/rlai.cs.ualberta.ca/RLAI/RLAIcourse/RLAIcourse2006.html
http://web.stanford.edu/class/cs234/schedule.html

University of Michigan - Ann Arbor

The fundamental concepts that form the basis of reinforcement learning are:

 - Agent
 - Environment
 - Action
 - State
 - Reward

The agent learns through repeated interaction with the environment. To be successful, the agent needs to:
 - Learn the interaction between states, actions, and subsequent rewards.
 - Determine which action will provide the optimal outcome.

Fundamental Concepts for Reinforcement Learning

Here is a useful website for starters to understand reinforcement learning: https://missinglink.ai/guides/deep-learning-healthcare/tensorflow-reinforcement-learning-introduction-and-hands-on-tutorial/

Useful Website for Reinforcement learning

The environment uses information about the agent’s current state and action as input and returns the agent’s reward and its next state as output. An example of an environment is the laws of physics. To the agent, the environment is a black box.

Environment in Reinforcement Learning

A state is a situation in which the agent finds itself. It can be a specific place and moment, a situation returned by the environment or any future situation. This includes the set of actions available, as well as other considerations such as tools, dangers, or rewards.

State in Reinforcement Learning

The entity that executes actions. For example, a robot deciding on a path to walk, or a trader deciding what to buy or sell. The agents must be able to understand the environmental state and execute actions accordingly, which would alter the state of the environment. 

Agent in Reinforcement Learning

An action is the set of all possible moves the agent can make. Agents choose from a list of possible actions. In video games, for example, the list might include: running right or left, jumping high or low, crouching or standing still.

Action in Reinforcement Learning

A reward is a feedback used to effectively evaluate the agent’s action, can be immediate or delayed. Depending on the reward, the event can be considered good or bad. In case agent receives a low reward after performing particular action which was guided by the policy, then the policy would be modified.  The purpose of the agent is to maximize the reward.
 

Reward in Reinforcement Learning

Here is a free pdf version of a book about Reinforcement Learning:
http://incompleteideas.net/book/RLbook2020.pdf

Useful Book for Reinforcement Learning

Useful Tutorials about Math behind Reinforcement Learning

Reinforcement learning is a general approach to solve reward-based problem. The building of it is based on mathematical foundations. We can also view it as a sequential decision making and control problem mathematically. For example, when driving a car, we need to make decisions about turning left/right or going straight after the previous decisions. And the key ingredient in reinforcement learning is Markov Decision Process.

Math Behind Reinforcement Learning

It is very challenging in the reinforcement learning to find a perfect balance between using previously used effective action(**exploitation**) and exploring new ones (**exploration**).

In order to solve this problem, simple $\epsilon$-Greedy algorithm can be used, in which we with some small probability $\epsilon$ we would select action randomly (**exploration**) and with $1 - \epsilon$ probability use previously selected effective action(**exploitation**).  

Exploration/Exploitation trade-off

Classification of Reinforcement Learning Methods

On-policy methods attempt to evaluate or improve the policy that is used to make decisions. In contrast, off-policy methods evaluate or improve a policy different from that used to generate the data. Classicla on-policy methods include Sarsa Algorithm. As a comparison, Q-learning method is a classical off-policy method.

On-policy vs Off-policy

Compared to value based methods, actor-critic methods focus on modeling the probability distribution of policies. For value-based methods, the logic is more like a greedy algorith, we want to maximize the value function for every step. As a result, we need to search through the action space to get the best action. But in actor-critic method, they are like using actor to perform actions based on the probability distribution of policies and using critics to judge the performance and adjust the probability distribution. They don't directly always choose the best action. 

Actor-Critic Methods

van Hasselt, H., Guez, A., & Silver, D. (2016). Deep Reinforcement Learning with Double Q-Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1). Retrieved from https://paperswithcode.com/paper/deep-reinforcement-learning-with-double-q

Deep Reinforcement Learning with Double Q-learning

One of the most popular reinforcement learning algorithms, known to sometimes learn unrealistically high action values since it includes a maximization step that tends to prefer overestimated values.

Q-learning

Borges, Alexandre and Arlindo L. Oliveira. “Combining Off and On-Policy Training in Model-Based Reinforcement Learning.” ArXiv abs/2102.12194 (2021): n. pag.
https://arxiv.org/pdf/2102.12194.pdf

Combining Off and On-Policy Training in Model-Based
Reinforcement Learning

MuZero is a model-based reinforcement learning method which contains a representation function, a dynamics function, and a prediction function.The representation function encodes an observation sequence into a hidden state. The dynamics function evaluates the current hidden state and a potential action to determine the potential next hidden state and a reward value. Lastly, the prediction function is the policy generation function. They use similar parameters and are optimized according to a combined loss function.

A Monte Carlo Tree Search (MCTS) is often used when training to evaluate action trees from the current/ root state.

Learn Before

Related