Case context: An engineer is applying reinforcement learning to fly a helicopter through a trajectory T that ends in a landing. They design a reward function R(.) that gives a large negative reward, such as R(T) = -1,000, if the helicopter crashes, and a positive reward for a safe landing. However, the current reward function only checks whether the helicopter crashed or landed safely, and does not account for how bumpy the landing was, whether the helicopter landed at the desired spot, or how rough the ride was for passengers.

Question: What should the engineer diagnose as the main limitation of this reward function, and how might they revise it to better reflect what makes a trajectory genuinely desirable?

Sample answer: The engineer should diagnose that the reward function is too simplistic because it only distinguishes between crashing and landing safely, without accounting for other factors that determine whether a trajectory is truly desirable. To improve it, the reward function should also incorporate how smooth or bumpy the landing was, whether the helicopter landed exactly at the intended spot, and how rough the ride was for passengers, combining these factors into a single score that better reflects overall trajectory quality.

Key points:

• Reward function currently only distinguishes crash vs. safe landing
• Good reward functions must trade off bumpiness, landing accuracy, and ride roughness
• Designing good reward functions is difficult because multiple factors must be balanced
• Revised reward function should combine these factors into a single score

Rubric: Full credit: identifies that the reward function ignores bumpiness, landing accuracy, and ride roughness, and proposes incorporating these factors into the score. Partial credit: identifies the limitation but does not propose a revision, or vice versa. No credit: does not identify the missing trade-offs.