What two practical bottlenecks limit endless growth in model size and training data?
Question: In one to three sentences, name two practical bottlenecks that make it unrealistic to keep increasing a neural network's size and its training data forever.
Sample answer: One bottleneck is that very large models take much longer to train, so compute and time become limiting factors. The other is that additional training data may eventually be unavailable, too expensive, or too difficult to collect.
Key points:
- Slow, expensive training of larger models
- Limited availability or high cost of additional data
Rubric: Full credit requires identifying both the compute/time bottleneck and the data-availability bottleneck.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
What is the main practical limit on repeatedly making a neural network much larger?
True or False: With enough model capacity and enough labeled data, a learning system can in principle achieve strong results on many tasks.
Training a very large model is _____, so increasing model size has practical limits.
Match each scaling approach with the limitation it eventually runs into.
Order the steps that explain why model and data scaling have practical limits.
Why Infinite Scaling Is an Idea, Not a Plan
Practical limits when scaling a fraud model
What two practical bottlenecks limit endless growth in model size and training data?
What can stop a team from relying on ever more training data?
True or False: Making a neural network much larger always makes training faster and removes compute bottlenecks.