Estimating Variance with a Held-Out Same-Source Set
A classifier is fit on 8,000 labeled examples from one source. Another 800 examples from that same source are set aside and never used for fitting. The difference between the training error and the _____ error is used to estimate variance on the training-set distribution.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
What does a 1% gap between training error (14%) and training-dev error (15%) suggest?
True or False: An algorithm with 8% training error, 9% training-dev error, and 18% dev error is showing high variance on the training-set distribution.
High avoidable bias with a distribution shift
Match each error gap in the 8%/9%/17% scenario to the machine learning problem it diagnoses.
Order the diagnostic steps for identifying high bias and data mismatch without high variance.
Diagnosing Bias, Variance, and Distribution Shift from Error Rates
True or False: If training error is 8%, training-dev error is 9%, and dev error is 15%, the mismatch between training-dev and dev explains more of the drop than variance does.
Estimating Variance with a Held-Out Same-Source Set
Match each diagnosis to the evidence in the 12%/13%/19% scenario.
Order the reported error rates from lowest to highest in a case where the model fits the training set well but struggles on a shifted dev set.
Interpreting training, in-domain, and deployment errors
Diagnosing Bias and Distribution Shift
What problem is ruled out?