Learn Before
Strong Stages, Weak End-to-End System
If every stage of a multi-step learning system is roughly at human level, but the whole system is still much worse than human performance, the system design is usually the problem and should be redesigned rather than adjusted only stage by stage.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Far-Behind Components Are Good Improvement Targets
Strong Stages, Weak End-to-End System
What information should a human evaluator see when judging the Route Planning component in a delivery-robot pipeline?
Checking a subsystem against human performance is mainly a formal, rigid debugging procedure.
How far is a component from human performance?
Match each autonomous delivery robot component to its primary output or function.
Order the informal debugging questions for a package-delivery drone system.
Why should the human reviewer for a parcel-routing module use only the classifier's outputs instead of the warehouse camera feed?
In a warehouse robot pipeline, both the shelf-item detector and the obstacle detector can send information directly to the route planner.
A fair human comparison uses the same inputs
Match each diagnostic question to the part of a medical triage pipeline it evaluates.
Diagnosing a Multi-Stage Perception Pipeline
Why Human Comparisons Must Use the Same Component Inputs
Comparing a Delivery Robot Planner to Human Performance
Fair Human Comparison for a Route-Planning Module
Learn After
When a Processing Stage Lacks the Right Input
If every component of a machine learning system matches human performance, but the full system is still much worse than human performance, what should you conclude?
Perfect Subsystems Always Produce a Perfect Pipeline
If every stage of a learning system looks strong on its own but the full system still underperforms, the overall design is usually _____ and should be rethought.
Match each pipeline diagnosis idea to its meaning.
Order the diagnostic steps for checking whether a multi-stage model is poorly designed even when each stage seems strong on its own.
In a warehouse-robot example, people who see only the robot's camera feeds choose better routes than the full machine-learning pipeline, even though each module has been tuned to human level. What is the best conclusion?
Error analysis on a weak machine learning pipeline can reveal that the overall system design should be changed.
When comparing a system module to human performance, the human should receive the _____ that the module receives.
Match each system outcome to the most reasonable diagnosis.
Put these steps in the order that shows why a model pipeline should be redesigned when every component is already performing at human level.
What a Strong Pipeline Can Still Get Wrong
Troubleshooting an Autonomous Delivery System
What to Do When an End-to-End System Is Still Weak