Google

For a reinforcement learning system whose trajectory is worse than a human pilot trajectory, compare the reward assigned to the human trajectory with the reward assigned to the algorithm trajectory. If the human trajectory scores higher, improving the reinforcement learning algorithm is worthwhile; if it does not, improve the reward function.

Optimization Verification for Reinforcement Learning Reward Functions

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

Applying Optimization Verification in RL

Diagnosing Reward Function Issues

Interpreting _____ in Optimization Verification

Matching Optimization Verification Components

Executing the Optimization Verification Test

Question:
Explain the significance of the inequality R(Thuman) > R(Tout) in the context of the Optimization Verification test. What does it tell you about the reward function and the reinforcement learning algorithm?

Sample answer:
When the inequality R(Thuman) > R(Tout) holds, it means that the reward function correctly assigns a higher score to the superior human trajectory compared to the inferior trajectory generated by the algorithm. This signifies that the reward function is functioning as intended, accurately reflecting the desired tradeoff or behavior. Consequently, the fault lies with the reinforcement learning algorithm, which is failing to maximize the reward and is instead settling for an inferior trajectory. The next step is to improve the learning algorithm's optimization process.

Key points:
- Validates the reward function
- Identifies the learning algorithm as the problem
- Indicates the algorithm is failing to maximize the reward

Rubric:
A strong answer should clearly state that the inequality validates the reward function and identifies the learning algorithm as the source of the poor performance.

Analyzing the R(Thuman) > R(Tout) Inequality

Case context:
You are building an RL agent to land a simulated helicopter. You define a reward function to balance landing accuracy and ride smoothness. The agent learns to land, but its trajectory (Tout) is much bumpier than a human pilot's trajectory (Thuman). You apply the Optimization Verification test and find that R(Thuman) is less than R(Tout).

Question:
Based on this finding, what specific component of your system should you focus on improving, and why?

Sample answer:
You should focus on improving the reward function. Because R(Thuman) is less than R(Tout), the reward function is incorrectly assigning a higher score to the bumpier, inferior algorithm trajectory than to the smoother, superior human trajectory. This means the reward function fails to specify the ideal tradeoff between ride bumpiness and landing accuracy, so it must be redesigned.

Key points:
- Improve the reward function
- The function assigns higher reward to the inferior trajectory
- The tradeoff specification is flawed

Rubric:
The answer must identify the reward function as the component to improve and explain that it is incorrectly scoring the trajectories.

Helicopter Landing Reinforcement Learning

Question:
According to the Optimization Verification test, under what specific condition should you deduce that your reinforcement learning algorithm (and not the reward function) needs improvement?

Sample answer:
You should improve the reinforcement learning algorithm if the reward assigned to the superior human trajectory is strictly greater than the reward assigned to the algorithm's inferior trajectory.

Key points:
- Compare human and algorithm trajectories
- Human trajectory scores higher
- Indicates algorithm fails to optimize

Rubric:
The response must mention comparing the human and algorithm trajectories and checking if the human trajectory scores higher.

Learn Before

Related