Why Infinite Scaling Is an Idea, Not a Plan
Question: Write a short essay explaining why the idea that a model could keep getting better forever by adding more capacity and more data does not translate into ordinary machine learning projects.
Sample answer: If model capacity and training examples could grow without any limit, many problems would likely become easier and better performance would seem attainable. Real projects do not have that freedom. Very large models take much longer to train and require far more computing power, and datasets are not unlimited either. At some point, there is no more useful data to collect. Because of these constraints, engineers must improve systems within practical limits instead of depending on endless growth.
Key points:
- Unlimited capacity and unlimited data create an appealing theoretical picture
- Large models are expensive and slow to train, which creates computing constraints
- Data availability eventually runs out, so scaling cannot continue forever
- Real-world practice is limited by these constraints rather than the idealized picture
Rubric: Full credit requires stating the idealized claim, identifying both real-world limits (computing cost and limited data supply), and clearly explaining why those limits separate theory from practice.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
What is the main practical limit on repeatedly making a neural network much larger?
True or False: With enough model capacity and enough labeled data, a learning system can in principle achieve strong results on many tasks.
Training a very large model is _____, so increasing model size has practical limits.
Match each scaling approach with the limitation it eventually runs into.
Order the steps that explain why model and data scaling have practical limits.
Why Infinite Scaling Is an Idea, Not a Plan
Practical limits when scaling a fraud model
What two practical bottlenecks limit endless growth in model size and training data?
What can stop a team from relying on ever more training data?
True or False: Making a neural network much larger always makes training faster and removes compute bottlenecks.