Essay

Why Infinite Scaling Is an Idea, Not a Plan

Question: Write a short essay explaining why the idea that a model could keep getting better forever by adding more capacity and more data does not translate into ordinary machine learning projects.

Sample answer: If model capacity and training examples could grow without any limit, many problems would likely become easier and better performance would seem attainable. Real projects do not have that freedom. Very large models take much longer to train and require far more computing power, and datasets are not unlimited either. At some point, there is no more useful data to collect. Because of these constraints, engineers must improve systems within practical limits instead of depending on endless growth.

Key points:

  • Unlimited capacity and unlimited data create an appealing theoretical picture
  • Large models are expensive and slow to train, which creates computing constraints
  • Data availability eventually runs out, so scaling cannot continue forever
  • Real-world practice is limited by these constraints rather than the idealized picture

Rubric: Full credit requires stating the idealized claim, identifying both real-world limits (computing cost and limited data supply), and clearly explaining why those limits separate theory from practice.

0

1

Updated 2026-08-12

Contributors are:

Who are from:

Tags

Machine Learning

Deep Learning

Supervised Learning

Dive into Deep Learning @ D2L

Data Science

Machine Learning Strategy

Machine Learning Yearning @ DeepLearning.AI