Comparing Errors Across Two Data Sources
A diagnostic error table can help you compare a model on two different data sources, such as recordings from quiet rooms and recordings from noisy rooms. Put the two sources across the top of the table and use three rows for human-level error, error on examples used for training, and error on examples the model never saw during training. This layout makes it easier to tell whether a problem comes from one source, from the training data, or from generalization. Filling in more entries can sometimes uncover patterns that are hard to notice from a single summary number.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Do Not Randomly Mix Sources When Building Validation and Test Sets
Use a Small Amount of Target-Distribution Data in Training with Auxiliary Data
When to Reduce the Influence of Extra Training Data
Training-Development Set
Comparing Errors Across Two Data Sources
Training–Dev Distribution Gap
Domain adaptation solves only certain distribution-shift problems
Choosing Development and Test Data for a Specific Application
Factory Photos and Inspection Tablet Images as a Distribution Mismatch Example
Random Train/Test Splits Can Be Misleading When Deployment Data Differs
Which data should define the dev and test sets for the plant-disease app?
Development and test examples should be drawn to match the kind of data the deployed system is expected to see.
Development and test sets should match the kind of data you expect later.
Match each data group for the plant-disease app with its role.
Order the dataset choices when training data and evaluation data come from different sources.
Why training data and evaluation data may come from different sources
Find the evaluation-set mismatch in a voice-command detector.
Why did strong desktop-photo results not predict mobile-upload success?
Choosing Training and Evaluation Data from Two Image Sources
If training uses extra web-sourced photos, the dev and test sets must also contain web-sourced photos.
Learn After
In a comparison table for a speech recognizer, what belongs on the x-axis?
In the error table, the horizontal axis shows three different data distributions.
The diagnostic table compares two data _____ across the top and lists three error categories down the side.
Match each table axis or row label to its role in comparing errors across two data sets.
Order the steps for building a comparison table for two data groups.
What is one advantage of adding more entries to a diagnostic error table?
A table can make it easier to compare different kinds of errors across training and development sets.
One row in an error table reports performance on examples the model has _____.
Match each table comparison to the diagnostic insight it provides.
Steps for Interpreting an Error Table Across Two Data Sources
Interpreting a Performance Error Table
Building an error table for a model evaluated on two image streams.
Name the three error rows in the error table