Google

In reinforcement learning, the machine goes through trial and error processes where it is rewarded/penalized for the actions it performs. The machine's goal is to maximize the total reward by leveraging previous attempts to make the next decision. The model starts from completely random trials and eventually leads to something with very sophisticated tactics and skills. Compared to supervised/unsupervised learning, reinforcement learning is much more advanced. A reinforcement model will continuously learn, unlike supervised/unsupervised models which all have an endpoint after the training and test data phases.

Reinforcement Learning

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

A good way to understand reinforcement learning would be to think about how autonomous vehicles work. Instead of programming all of the rules it should follow (such as road signs, pedestrians, right of way, etc), it is much more easier and efficient to implement a reward/penalty system where the car learns all of the rules that it is supposed to abide by. Furthermore, it would be unreasonable to program individual rules because accidents can happen in unpredictable ways and rules/regulations are always changing and varies in different parts of the world. So, it is much better for autonomous cars to be always learning better ways to make decisions.

Reinforcement Learning Example - Autonomous Vehicles

Video games often involve completing levels, with increasing difficulty as you go through the game. In each level, you go in completely blind and make decisions that will either lead to you passing that level or failing. If you fail, you will now understand more about that particular level that will influence your decisions when you retry. In other words, you will understand what not to do, and try another approach. If you pass, you will understand what to do in future levels. This process is repeated until success. This is very similar to how reinforcement learning works. Rewards are given for successes, indicating that the decision(s) made are correct and should be learned for future use. Penalties are given for a failure, indicating that the decision(s) made should not be retried and instead a new approach should be used.

Reinforcement Learning Analogy - Video Games

Theobald, O. (2017). Machine learning for absolute beginners: a plain English introduction (2nd ed.). Scatterplot Press.

Online pdf: http://bmansoori.ir/book/Machine%20Learning%20For%20Absolute%20Beginners.pdf

Machine Learning for Absolute Beginners

Deep learning analyses a training set, identifies complex patterns and applies them to new data. 

Reinforcement learning works sequentially in an unknown environment━taking an action, evaluating the rewards, and adjusting the following actions accordingly.

Deep learning and reinforcement learning complement each other:

 - Reinforcement learning algorithms manage the sequential process of taking an action, evaluating the result, and selecting the next best action. However, they need a good mechanism to select the best action based on previous interactions.
 - Deep learning can be that mechanism━it is the most powerful method available today to learn the best outcome based on previous data.

 

Deep Learning vs. Reinforcement Learning

The fundamental concepts that form the basis of reinforcement learning are:

 - Agent
 - Environment
 - Action
 - State
 - Reward

The agent learns through repeated interaction with the environment. To be successful, the agent needs to:
 - Learn the interaction between states, actions, and subsequent rewards.
 - Determine which action will provide the optimal outcome.

Fundamental Concepts for Reinforcement Learning

(Encourage to continuously enrich and edit this node for sharing papers and their proposed useful ideas.)
1.Learning Action Representations for Reinforcement Learning Yash Chandak, Georgios Theocharous, James E. Kostas, Scott M. Jordan, Philip S. Thomas 

Reinforcement Learning Refrence and Cutting-edge Ideas

A team is developing a program to play a complex board game against human opponents. The program has no pre-existing data of past games to learn from. Instead, it is designed to learn by playing against itself repeatedly. After each game, the program receives a positive signal if it wins and a negative signal if it loses. Over time, it is expected to discover winning strategies on its own. Which of the following statements best analyzes why this learning approach is suitable for this task?

This scenario describes a classic reinforcement learning problem. Identify the Agent, Environment, State, Action, and Reward within this context. Then, explain how the reward structure incentivizes the agent to achieve the desired outcome of finding the exit quickly.

Robot Maze Navigation Strategy

A company is developing a system to recommend personalized news articles. The system's goal is to continuously adapt to a user's evolving interests. Two strategies are proposed:

**Strategy A:** Collect a massive, one-time dataset of user profiles and the articles they clicked on in the past. Train a model on this static dataset to predict which articles a new user is likely to click.

**Strategy B:** Design a system that learns through direct interaction. For each article it recommends, it receives a positive signal if the user clicks it and a negative signal if they ignore it. The system's objective is to maximize the positive signals it receives over time.

Evaluate these two strategies. Which one is better suited for creating a system that adapts to a user's *evolving* interests? Justify your evaluation by comparing the core mechanics of each strategy and their implications for long-term personalization.

Evaluating Learning Strategies for a Recommendation System

In reinforcement learning for helicopter control, a reward function scores how good each possible trajectory is. The reward may penalize crashes heavily and reward safe landings, while trading off smoothness, landing location, ride roughness, and other desiderata.

Reward Function for Reinforcement Learning Trajectories

What is the machine's primary goal during the trial-and-error process in reinforcement learning?

Reinforcement learning models keep learning continuously, unlike supervised or unsupervised models.

In reinforcement learning, the machine aims to maximize the total _____ by using lessons from previous attempts.

Match each reinforcement learning term to its correct description.

Order the steps of applying reinforcement learning to a task like flying a helicopter.

Question:
Explain why reinforcement learning is described as more advanced than supervised or unsupervised learning. In your answer, address the role of trial and error, the reward function, and the concept of continuous learning.

Sample answer:
Reinforcement learning is considered more advanced because the machine is not simply given labeled data or asked to find patterns in unlabeled data; instead, it must learn through trial and error, receiving rewards or penalties for its actions and using that feedback to improve future decisions. This requires the system to develop increasingly sophisticated tactics and skills starting from completely random behavior. Additionally, unlike supervised or unsupervised models that stop learning once training and testing are complete, reinforcement learning models continue learning indefinitely, adapting as they encounter new situations. This ongoing, feedback-driven adaptation makes reinforcement learning more complex to design and implement than the fixed-endpoint nature of supervised or unsupervised learning.

Key points:
- Machine learns via trial and error, not from fixed labeled/unlabeled data
- Actions are rewarded or penalized to guide future decisions
- Model progresses from random trials to sophisticated tactics
- Reinforcement learning continues learning indefinitely
- Supervised/unsupervised models stop learning after training and test phases

Rubric:
Full credit: clearly explains trial-and-error learning, the role of reward/penalty feedback, and the continuous (non-endpoint) nature of reinforcement learning compared to supervised/unsupervised learning. Partial credit: mentions only one or two of these elements. No credit: fails to reference reward-based feedback or continuous learning.

Why is reinforcement learning considered more advanced than supervised or unsupervised learning?

Case context:
An engineer is applying reinforcement learning to fly a helicopter through a trajectory T that ends in a landing. They design a reward function R(.) that gives a large negative reward, such as R(T) = -1,000, if the helicopter crashes, and a positive reward for a safe landing. However, the current reward function only checks whether the helicopter crashed or landed safely, and does not account for how bumpy the landing was, whether the helicopter landed at the desired spot, or how rough the ride was for passengers.

Question:
What should the engineer diagnose as the main limitation of this reward function, and how might they revise it to better reflect what makes a trajectory genuinely desirable?

Sample answer:
The engineer should diagnose that the reward function is too simplistic because it only distinguishes between crashing and landing safely, without accounting for other factors that determine whether a trajectory is truly desirable. To improve it, the reward function should also incorporate how smooth or bumpy the landing was, whether the helicopter landed exactly at the intended spot, and how rough the ride was for passengers, combining these factors into a single score that better reflects overall trajectory quality.

Key points:
- Reward function currently only distinguishes crash vs. safe landing
- Good reward functions must trade off bumpiness, landing accuracy, and ride roughness
- Designing good reward functions is difficult because multiple factors must be balanced
- Revised reward function should combine these factors into a single score

Rubric:
Full credit: identifies that the reward function ignores bumpiness, landing accuracy, and ride roughness, and proposes incorporating these factors into the score. Partial credit: identifies the limitation but does not propose a revision, or vice versa. No credit: does not identify the missing trade-offs.

Diagnose a poorly designed reward function for an autonomous helicopter landing task.

Question:
In one to three sentences, explain why designing a good reward function for a task like helicopter flight is not easy.

Sample answer:
Designing a good reward function is difficult because it must balance multiple competing factors, such as how bumpy the landing was, whether the helicopter landed exactly at the desired spot, and how rough the ride was for passengers. Since the reward function is typically chosen by hand, capturing all of these trade-offs in a single score is challenging.

Key points:
- Reward function is typically chosen by hand
- Must trade off bumpiness, landing spot accuracy, and ride roughness
- Balancing multiple factors in one score makes design difficult

Rubric:
Full credit: mentions that the reward function must trade off multiple factors (e.g., bumpiness, landing accuracy, ride roughness) and is chosen by hand. Partial credit: mentions only that it's hand-designed or only one trade-off factor.

Learn Before

Related