Case context: A team trains a helicopter control agent using a reward function that gives R(T) = -1,000 for crashes and a positive reward for safe landings, with the exact positive value depending on landing smoothness. After training, the agent achieves high average reward across episodes, but engineers notice landings are frequently occurring far from the desired landing spot, even though crashes are rare.

Question: Based on the source material, what is the likely issue with this reward function, and how might it be revised to address it?

Sample answer: The likely issue is that the reward function heavily penalizes crashes and rewards smooth landings but does not adequately account for whether the helicopter lands at the desired spot. Since the reward function must trade off crash avoidance, landing smoothness, landing location, and ride roughness, if landing location is underweighted or omitted, the agent has little incentive to land at the correct spot as long as it avoids crashing and lands smoothly. The team should revise R(.) to include a term that penalizes distance from the desired landing location.

Key points:

• The reward function must trade off crash risk, landing smoothness, landing location, and ride roughness
• High average reward does not guarantee all desiderata are well balanced
• Landing location appears underweighted given the observed behavior
• A fix involves explicitly incorporating landing location into R(.)

Rubric: Full credit for identifying that landing location is a required tradeoff factor per the source and that underweighting it explains the observed behavior, plus a reasonable proposed fix; partial credit for a vague diagnosis without connecting to the tradeoff factors.