Google

In reinforcement learning for autonomous aircraft control, a reward function assigns a score to each candidate trajectory. It can penalize crashes sharply and reward successful landings, while also balancing factors such as landing accuracy, motion smoothness, passenger comfort, and other desired outcomes.

Reward Scores for Reinforcement Learning Trajectories

What Does the Reward Function Represent in a Delivery-Drone Example?

A smooth landing path can earn what kind of reward?

Match each delivery outcome to the usual reward assigned in a robot courier task.

Put the reward-design process for a drone delivery task in order.

Question:
In one to three sentences, explain why building a reward function for the helicopter task is challenging and what kinds of tradeoffs it must capture.

Sample answer:
It is difficult because one reward value has to summarize several goals at once. For a helicopter, the designer may want to avoid crashes, keep the flight smooth, land close to the target, and reduce discomfort, but improving one of those goals can make another worse. Because people usually choose the reward formula by hand, it is hard to assign weights that reflect all of those priorities correctly.

Key points:
- One reward must combine several goals
- The goals can conflict with one another
- The function is usually hand-specified
- Choosing the right weights is difficult

Rubric:
Full credit for explaining that the reward must balance multiple competing objectives, that it is manually designed, and that this makes correct specification hard; partial credit for naming only one objective.

Why Is It Hard to Design a Reward Function for a Helicopter?

Case context:
A team trains a mobile robot to return to a charging station. The reward gives a large penalty for collisions and a bonus for stopping smoothly, with a small extra reward whenever the robot ends an episode without getting stuck. After training, the average return is high, but operators notice that the robot often stops in the wrong charging zone several meters away from the dock.

Question:
What is the most likely flaw in the reward design, and how should the team update it?

Sample answer:
The reward appears to emphasize avoiding collisions and stopping smoothly, but it does not give enough importance to reaching the correct docking bay. A reward function should balance all of the goals that matter for the task, and here the location of the final stop is underweighted or missing. The fix is to add an explicit term that rewards being close to the target dock, or penalizes distance from it, so the agent has incentive to end in the right place.

Key points:
- The reward must balance safety, smooth stopping, and accurate final position
- A high average reward can hide an important missing objective
- The observed behavior suggests docking location is underweighted
- A distance-based term for the target dock would address the problem

Rubric:
Full credit for identifying that the final position is not sufficiently represented in the reward and for proposing a clear location-based correction; partial credit for noting that the reward is incomplete without explaining the tradeoff.

Why a warehouse robot scores well but still misses the docking bay

Question:
When designing a helicopter landing reward, what are two factors other than avoiding crashes that the reward should balance?

Sample answer:
Two valid factors are how gently the helicopter touches down and how close it lands to the target location. Minimizing passenger discomfort from a rough descent is also acceptable.

Key points:
- Gentle or smooth landing
- Accuracy of the landing position
- Low roughness or discomfort during descent

Rubric:
Full credit for naming any two of: smooth landing, accurate landing location, low roughness/discomfort; partial credit for one correct factor.

Learn Before

Related