Data availability is a major driver of recent deep learning progress because digital-device activity creates large amounts of data that can be fed to learning algorithms.

Data Availability as a Driver of Deep Learning Progress

Computational scale is a major driver of recent deep learning progress because it makes it possible to train neural networks large enough to take advantage of huge datasets.

Computational Scale as a Driver of Deep Learning Progress

Older learning algorithms such as logistic regression can plateau as more data is added, meaning their learning curves flatten and the algorithms stop improving even with more data.

Older Learning Algorithms Can Plateau with More Data

Larger neural networks can obtain better performance, and the best performance comes from training a very large neural network while also having a huge amount of data.

Large Neural Networks Benefit from Huge Data

In a small data regime, traditional algorithms may or may not do better depending on how features are hand-engineered. With only about 20 training examples, feature hand-engineering can matter more than choosing logistic regression or a neural network.

Small Data Regime Can Favor Hand-Engineered Features

According to Andrew Ng, what are the two biggest drivers of recent deep learning progress?

True or False: Neural network ideas are a brand-new invention of the last decade.

Two of the biggest drivers of recent progress have been data availability and _____.

Match each driver of deep learning progress to its correct description.

Order the reasoning steps explaining why deep learning is taking off now.

Question:
In a concise essay, explain the relationship between the age of deep learning ideas and the recent surge in their real-world impact, referencing the two drivers Ng identifies.

Sample answer:
Deep learning's core ideas, such as neural networks, have existed for decades but lacked the resources to reach their potential. Recent progress is not due to new theoretical breakthroughs but to two enabling factors: data availability, meaning far more labeled data now exists to train models, and computational scale, meaning far more processing power is available to train larger models on that data. Together, these two factors allowed old ideas to finally demonstrate strong performance, explaining why deep learning is 'taking off now' rather than decades ago.

Key points:
- Deep learning ideas are decades old
- Progress is driven by data availability and computational scale, not new ideas
- Both factors work together to enable performance gains
- This explains why deep learning is only now taking off

Rubric:
Full credit requires identifying that the ideas are old, naming both data availability and computational scale as drivers, and explaining how their combination enabled progress. Partial credit for naming only one driver or omitting the historical context.

Explain why decades-old deep learning ideas are only now driving major progress.

Case context:
A company trains two systems on the same growing dataset: an older learning algorithm and a large neural network. As the dataset grows from thousands to millions of examples, the older algorithm's accuracy stops improving, while the neural network's accuracy keeps rising with more data and more computational resources.

Question:
Using the concept of scale as a driver of deep learning progress, diagnose why these two systems behave differently as data increases.

Sample answer:
The older algorithm plateaus because it has a limited capacity to exploit additional data, so beyond a certain point extra examples do not translate into better performance. The large neural network, however, has the capacity to keep learning from more data, and its improvement also depends on having enough computational scale to train on that growing dataset. This illustrates the parent concept that recent progress in deep learning is driven jointly by data availability and computational scale, and that large neural networks are specifically positioned to benefit from both in ways older algorithms are not.

Key points:
- Older algorithms plateau despite more data
- Large neural networks continue improving with more data
- Computational scale is required to train on growing datasets
- This contrast illustrates the two drivers of deep learning progress

Rubric:
Full credit identifies that older algorithms plateau due to limited capacity, that neural networks scale with data, and that computational scale is necessary to realize that scaling. Partial credit for only describing the plateau or only describing the neural network's growth without connecting both drivers.

Diagnose why a company's old algorithm and new neural network perform differently as data grows.

Question:
Based on the concept that scale drives deep learning progress, briefly explain why hand-engineered features can outperform large neural networks when data is scarce.

Sample answer:
When data is scarce, large neural networks lack enough examples to learn useful patterns on their own, so their advantage from scale is not available. Hand-engineered features inject human domain knowledge directly into the model, compensating for the missing data and often yielding better performance in this small-data regime.

Key points:
- Neural networks rely on scale to perform well
- Small data limits the benefit of scale
- Hand-engineered features encode human knowledge
- This can outperform neural networks in low-data settings

Rubric:
Full credit for stating that neural networks need large data to benefit from scale and that hand-engineered features substitute human knowledge when data is limited. Partial credit for a vague or incomplete explanation.

In one to two sentences, why might a small dataset favor hand-engineered features over a large neural network?

Which statement best reflects the relationship between deep learning ideas and their timing of impact?

True or False: Data availability and computational scale are described as the two biggest drivers of recent deep learning progress.

Recent progress in deep learning has been driven by data availability and computational scale.

Google

Machine learning is a field that studies how computer systems can utilize experience—typically in the form of data—to improve their performance on specific tasks. It integrates principles from statistics, data mining, and optimization, and is frequently employed as a primary method for implementing artificial intelligence solutions.

Machine Learning

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

Representational learning is a class of machine learning that focuses on automatically discovering the most appropriate way to represent data. By learning these representations directly from the data, it eliminates the need for manual feature engineering and allows models to effectively process and understand complex inputs.

Representational Learning

Supervised learning learns a function mapping inputs x to outputs y using labeled training examples (x,y). Linear regression, logistic regression, and neural networks are examples of supervised learning algorithms.

Supervised Learning

Kaggle is a popular online platform that hosts machine learning competitions. Each competition centers on a specific dataset, and many are sponsored by stakeholders who offer prizes for the winning solutions. The platform enables direct quantitative comparisons of competing approaches and fosters collaboration by allowing users to interact via forums and share code.

Kaggle Platform

Combining classical machine learning and deep learning within predictive analytics platforms enables users to forecast future trends and behaviors with high precision. These data-driven insights accelerate time-to-value across industries by optimizing courses of action and reducing margins of error, effectively replacing retrospective interpretations with forward-looking decision-making.

Predictive Analytics for Accelerated Decision-Making

A development set, sometimes called a hold-out cross validation set, is used to tune parameters, select features, and make other decisions about the learning algorithm.

Development Set (Dev Set)

A test set is used to evaluate algorithm performance, but not to make decisions about which learning algorithm or parameters to use.

Test Set

When multiple evaluation metrics matter, such as accuracy and running time of a learning algorithm, one way to combine them is to define an acceptable value for one metric and then optimize another metric subject to meeting that criterion. In the passage's example, running time is the satisficing metric and accuracy is the optimizing metric.

Optimizing and Satisficing Metrics

For a new machine learning problem, it is very difficult to know in advance which approach will work best. Even experienced machine learning researchers usually try out many dozens of ideas before they discover something satisfactory.

Difficulty of Knowing the Best ML Approach in Advance

Machine learning is a highly iterative process. When building a machine learning system, the author often follows a loop: start with an idea about how to build the system, implement the idea in code, and carry out an experiment that tells how well the idea worked. Based on what is learned, generate more ideas and keep iterating; many dozens of ideas may be tried before finding one that is satisfactory.

Iterative Loop of Machine Learning Development

There are two major sources of error in machine learning: bias and variance. Understanding them helps decide whether adding data and other tactics to improve performance are a good use of time.

Bias and Variance as Two Major Sources of Error

End-to-end deep learning can directly learn outputs that are more complex than a number. With the right labeled input-output pairs, end-to-end learning can sometimes learn outputs such as a sentence, an image, audio, or other outputs richer than a single number.

End-to-End Learning

A pipeline may be flawed if the overall system falls far short of human-level performance even though each individual component is at or near human-level performance given its inputs. This suggests that the pipeline is missing information and should be redesigned.

Machine Learning Pipeline System

Sentiment classification is the problem of recognizing whether text expresses a positive or negative opinion, such as whether an online product review says the writer liked or disliked the product.

Sentiment Classification

Machine learning strategy is the practice of choosing useful directions for improving a machine learning application. It uses clues from the problem to decide what is useful to try and what is not useful to try.

Learn Before

Related

Learn After