Learn Before
Comparing a Delivery Robot Planner to Human Performance
Case context: You are building an indoor delivery robot. A camera-based detector finds people, carts, and boxes, and a route-planning module uses those detected object locations to choose a path through a hallway. You want to debug the route planner by comparing it with human performance. A teammate suggests showing volunteers the full security camera feed and asking whether they could plan a better route.
Question: Using the principles of informal pipeline debugging, explain why your teammate's idea is not a fair comparison and describe the correct way to set up the human test.
Sample answer: The proposal is not fair because the full camera feed gives the human more information than the planner receives. In the video, a person could use details such as wall edges, small obstacles, or motion cues that are not available to the planning module if it only sees detected object locations. The human test should instead use only the detector outputs, such as the positions of people, carts, and boxes, and ask the person to choose a route from that limited input. That way, the comparison measures how well a human performs with the same information the planner gets.
Key points:
- Full video contains extra information that the planner does not use.
- The human should see only the detector outputs, not the raw feed.
- The comparison should be made using identical inputs for the human and the component.
Rubric: Learners must explain that the raw video gives the human additional cues beyond the planner's inputs. They must also state that the proper evaluation gives the human only the outputs from the detection stage.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Far-Behind Components Are Good Improvement Targets
Strong Stages, Weak End-to-End System
What information should a human evaluator see when judging the Route Planning component in a delivery-robot pipeline?
Checking a subsystem against human performance is mainly a formal, rigid debugging procedure.
How far is a component from human performance?
Match each autonomous delivery robot component to its primary output or function.
Order the informal debugging questions for a package-delivery drone system.
Why should the human reviewer for a parcel-routing module use only the classifier's outputs instead of the warehouse camera feed?
In a warehouse robot pipeline, both the shelf-item detector and the obstacle detector can send information directly to the route planner.
A fair human comparison uses the same inputs
Match each diagnostic question to the part of a medical triage pipeline it evaluates.
Diagnosing a Multi-Stage Perception Pipeline
Why Human Comparisons Must Use the Same Component Inputs
Comparing a Delivery Robot Planner to Human Performance
Fair Human Comparison for a Route-Planning Module