Case context: You are building an RL agent to land a simulated helicopter. You define a reward function to balance landing accuracy and ride smoothness. The agent learns to land, but its trajectory (Tout) is much bumpier than a human pilot's trajectory (Thuman). You apply the Optimization Verification test and find that R(Thuman) is less than R(Tout).

Question: Based on this finding, what specific component of your system should you focus on improving, and why?

Sample answer: You should focus on improving the reward function. Because R(Thuman) is less than R(Tout), the reward function is incorrectly assigning a higher score to the bumpier, inferior algorithm trajectory than to the smoother, superior human trajectory. This means the reward function fails to specify the ideal tradeoff between ride bumpiness and landing accuracy, so it must be redesigned.

Key points:

• Improve the reward function
• The function assigns higher reward to the inferior trajectory
• The tradeoff specification is flawed

Rubric: The answer must identify the reward function as the component to improve and explain that it is incorrectly scoring the trajectories.