Question: In one to three sentences, explain why building a reward function for the helicopter task is challenging and what kinds of tradeoffs it must capture.

Sample answer: It is difficult because one reward value has to summarize several goals at once. For a helicopter, the designer may want to avoid crashes, keep the flight smooth, land close to the target, and reduce discomfort, but improving one of those goals can make another worse. Because people usually choose the reward formula by hand, it is hard to assign weights that reflect all of those priorities correctly.

Key points:

• One reward must combine several goals
• The goals can conflict with one another
• The function is usually hand-specified
• Choosing the right weights is difficult

Rubric: Full credit for explaining that the reward must balance multiple competing objectives, that it is manually designed, and that this makes correct specification hard; partial credit for naming only one objective.