Google

A computer vision system trained on synthetic car images from a video game will probably not work well even if the images look realistic. If 100,000 training examples all come from about 20 car designs, the system will overfit to those car designs and fail to generalize well to dev/test sets that include other car designs.

Overfitting to a Few Synthetic Car Designs

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

Why does a car-recognition system trained on 100,000 synthetic images from only ~20 car designs fail to generalize?

True or False: Realistic-looking synthetic images guarantee good generalization if enough images are generated.

If 100,000 training images all come from ~20 car designs, the system will _____ to those specific designs.

Match each dataset property in the synthetic car example to its description.

Order the reasoning steps explaining why the synthetic car training data leads to overfitting.

Question:
Explain why having 100,000 synthetic training examples does not prevent overfitting when those examples come from only about 20 car designs. In your answer, address the role of data diversity versus data quantity.

Sample answer:
Although 100,000 examples sounds like a large dataset, the true source of information is the underlying variety of car designs, not the raw image count. Because all images derive from only ~20 designs, the model is repeatedly exposed to the same underlying shapes and features with only minor rendering variation. This means the effective diversity of the training set is far smaller than 100,000 would suggest. Since dev/test sets reflect the full real-world distribution of car designs, a model trained on such a narrow set of designs learns features specific to those 20 cars rather than general car-recognition features, causing it to overfit and perform poorly on unseen designs.

Key points:
- 100,000 images is a large quantity but represents only ~20 underlying designs
- Diversity of underlying designs matters more than raw example count
- Dev/test sets reflect the full real-world distribution of car designs
- The mismatch between narrow synthetic diversity and broad real-world diversity causes overfitting

Rubric:
Full credit: explains that quantity of images does not equal diversity of underlying designs, notes the mismatch between the narrow training distribution and the broad dev/test distribution, and explains overfitting results from this mismatch. Partial credit: mentions overfitting but does not clearly distinguish quantity from diversity.

Explain why example count alone does not prevent overfitting in the synthetic car dataset.

Case context:
A team partners with a video game company to generate 100,000 realistic synthetic images of cars using the game's graphics models, which include about 20 distinct car designs. The team trains a computer vision model on these images and achieves very high accuracy on their training set, but the model performs poorly on a dev set containing photos of real cars on roads.

Question:
What is the most likely cause of the poor dev set performance, and what does this reveal about the relationship between training data volume and data diversity?

Sample answer:
The most likely cause is overfitting to the limited set of ~20 car designs present in the synthetic data. Even though the training set has 100,000 images and achieves high training accuracy, all those images share the same small set of underlying car shapes, so the model has effectively memorized features specific to those 20 designs rather than learning generalizable car-recognition features. The real-world dev set includes a much wider variety of car designs, so the model's narrow learned features fail to transfer. This reveals that having many training examples does not compensate for low underlying diversity; what matters is whether the training distribution adequately represents the distribution the model will be tested and deployed on.

Key points:
- High training accuracy with poor dev accuracy is a signature of overfitting
- The cause is limited diversity (~20 designs), not insufficient example count
- Real-world dev sets represent a much broader distribution of car designs
- Training data volume does not substitute for representative diversity

Rubric:
Full credit: correctly identifies overfitting to the 20 designs as the cause, distinguishes example quantity from design diversity, and connects this to the mismatch with the real-world dev set distribution. Partial credit: identifies overfitting but does not explain the quantity-versus-diversity distinction.

Diagnose why a car-recognition model performs poorly despite strong training accuracy.

Question:
In one to three sentences, describe what the ~20 car designs in the video game represent relative to the full real-world distribution of cars that a dev/test set is likely to include.

Sample answer:
The ~20 car designs represent only a minuscule fraction of the full real-world distribution of cars. While the synthetic images may look realistic, they fail to capture the vast variety of car shapes and models found on actual roads, which is what the dev/test set reflects.

Key points:
- ~20 designs is a minuscule fraction of all real-world car designs
- Realistic appearance does not equal representative diversity
- Dev/test sets reflect the broader real-world distribution

Rubric:
Full credit: states that the 20 designs are a small/minuscule subset of the real-world car distribution and notes the mismatch with dev/test data. Partial credit: mentions limited variety without connecting it to the dev/test set.

Learn Before

Related