Question: Using the helicopter example, explain why it is not easy to design good reward functions, and discuss the tradeoffs a reward function must balance.

Sample answer: Designing a good reward function is difficult because a single score R(T) must simultaneously capture many competing factors: whether the helicopter crashed, how bumpy the landing was, whether it landed at the exact desired spot, and how rough the ride was for passengers. These factors must be traded off against each other, and it is not obvious how to weight them numerically. Because the reward function is chosen by hand, the designer must anticipate all the ways a trajectory could be more or less desirable, which is inherently hard to get right.

Key points:

• A single score R(T) must capture multiple competing factors
• Factors include crash risk, landing smoothness, landing location, and ride roughness
• The reward function is typically chosen by hand
• Balancing tradeoffs among these factors is inherently difficult

Rubric: Full credit for explaining that multiple competing desiderata must be traded off in one score, that the function is hand-designed, and that this makes it hard to specify correctly; partial credit for mentioning only one factor.