Choosing Model and Compute Capacity for a Very Large Dataset
Case context: A team has collected 8 million labeled product images. They train a compact neural network on this data, but the training and validation performance stop improving even though the dataset is much larger than anything they used before.
Question: Using the idea that larger data can require larger computation and model capacity, what is the most likely bottleneck, and what should the team change?
Sample answer: The most likely bottleneck is that the network is too small to make use of the dataset. The team should increase model capacity by training a larger network and also provide enough computation to support training that larger model.
Key points:
- The current model is too small for the scale of the data.
- A larger network is needed.
- Additional compute is needed to train the larger model.
Rubric: The response must identify insufficient model size as the main limitation and recommend both a larger model and more computational resources.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
What is the main purpose of increased computational scale in deep learning?
A small gap between training error and training-dev error suggests the model generalizes similarly to unseen data from the same distribution.
Why larger compute can improve deep learning
Match each term to its role in explaining why larger compute can accelerate deep learning gains.
Put the steps in order: why more compute can improve deep learning results
When did it become practical to train neural networks large enough to benefit from very large datasets?
Computing power by itself is enough to explain recent deep learning progress.
Neural Network Size and Data
Match each development factor with the limitation it removes in modern deep learning.
Order the actions a practitioner would take when trying to use compute to benefit from a very large dataset.
How More Compute Helps When Training on Very Large Datasets
Choosing Model and Compute Capacity for a Very Large Dataset
Why More Compute Helps Deep Learning Use Large Data