Case Study

Comparing a Delivery Robot Planner to Human Performance

Case context: You are building an indoor delivery robot. A camera-based detector finds people, carts, and boxes, and a route-planning module uses those detected object locations to choose a path through a hallway. You want to debug the route planner by comparing it with human performance. A teammate suggests showing volunteers the full security camera feed and asking whether they could plan a better route.

Question: Using the principles of informal pipeline debugging, explain why your teammate's idea is not a fair comparison and describe the correct way to set up the human test.

Sample answer: The proposal is not fair because the full camera feed gives the human more information than the planner receives. In the video, a person could use details such as wall edges, small obstacles, or motion cues that are not available to the planning module if it only sees detected object locations. The human test should instead use only the detector outputs, such as the positions of people, carts, and boxes, and ask the person to choose a route from that limited input. That way, the comparison measures how well a human performs with the same information the planner gets.

Key points:

  • Full video contains extra information that the planner does not use.
  • The human should see only the detector outputs, not the raw feed.
  • The comparison should be made using identical inputs for the human and the component.

Rubric: Learners must explain that the raw video gives the human additional cues beyond the planner's inputs. They must also state that the proper evaluation gives the human only the outputs from the detection stage.

0

1

Updated 2026-08-12

Contributors are:

Who are from:

Tags

Machine Learning

Deep Learning

Supervised Learning

Dive into Deep Learning @ D2L

Data Science

Machine Learning Strategy

Machine Learning Yearning @ DeepLearning.AI