Case context: A company trains two systems on the same growing dataset: an older learning algorithm and a large neural network. As the dataset grows from thousands to millions of examples, the older algorithm's accuracy stops improving, while the neural network's accuracy keeps rising with more data and more computational resources.

Question: Using the concept of scale as a driver of deep learning progress, diagnose why these two systems behave differently as data increases.

Sample answer: The older algorithm plateaus because it has a limited capacity to exploit additional data, so beyond a certain point extra examples do not translate into better performance. The large neural network, however, has the capacity to keep learning from more data, and its improvement also depends on having enough computational scale to train on that growing dataset. This illustrates the parent concept that recent progress in deep learning is driven jointly by data availability and computational scale, and that large neural networks are specifically positioned to benefit from both in ways older algorithms are not.

Key points:

• Older algorithms plateau despite more data
• Large neural networks continue improving with more data
• Computational scale is required to train on growing datasets
• This contrast illustrates the two drivers of deep learning progress

Rubric: Full credit identifies that older algorithms plateau due to limited capacity, that neural networks scale with data, and that computational scale is necessary to realize that scaling. Partial credit for only describing the plateau or only describing the neural network's growth without connecting both drivers.