Widespread use of phones, computers, and other connected devices produces very large datasets, and that growing supply of data has helped recent deep learning systems improve.

Digital Activity Expands the Data Pool for Deep Learning

Computational scale is one of the main reasons modern deep learning has improved so quickly, because larger computing capacity made it feasible to train neural networks big enough to benefit from very large datasets.

Computational Scale and Recent Deep Learning Gains

Models such as logistic regression can reach a point where adding more labeled examples brings little or no extra improvement. At that stage, the learning curve becomes nearly flat, so extra data no longer changes performance much.

Some Simpler Models Stop Improving Much After More Data

Very large neural networks tend to perform best when they are trained on a very large amount of data.

Large Neural Networks Need Plenty of Data

When a dataset contains only a very small number of labeled examples, success depends heavily on how the inputs are represented. With roughly 20 training examples, careful feature design can matter more than choosing between logistic regression and a small neural network. Either model can work better or worse depending on those features.

Tiny Training Sets Make Feature Design Crucial

Which pair best explains the main forces behind recent deep learning gains?

True or False: Most neural-network ideas were invented only in the last ten years.

Recent progress has been driven by more data and more _____.

Match the main forces behind progress in deep learning to their descriptions.

Arrange the explanation for why deep learning has accelerated recently.

Question:
In a concise essay, explain why ideas such as neural networks have existed for a long time but only recently began producing major practical gains.

Sample answer:
Core deep learning ideas are not new; researchers have studied them for many years. What changed was the environment around them. Two major enabling factors made the ideas work well at scale: much larger amounts of labeled data became available, and much more computational power became accessible for training larger models. Once those two constraints loosened, older methods could finally be trained effectively and produce strong results in real applications.

Key points:
- The underlying ideas are old
- Progress comes from more data and more computing power
- The two factors work together to make larger, better-trained models possible
- This explains the recent rise in practical impact

Rubric:
Full credit requires stating that the ideas are longstanding, naming both data availability and computational scale as the drivers, and explaining how the two together enabled the recent progress. Partial credit for identifying only one driver or leaving out the historical context.

Why long-standing deep learning ideas became effective recently

Case context:
A healthcare startup trains two predictors on the same patient-record dataset. One is a simple linear classifier, and the other is a large neural network. The training set grows from 12,000 records to 3.2 million records. The linear classifier improves a little at first and then levels off, while the neural network keeps getting better as more records and more GPU training time are added.

Question:
Using the idea that scale drives progress in deep learning, explain why the two models respond differently as the dataset grows.

Sample answer:
The linear classifier reaches a point where extra records do not help much because its representational power is limited. In contrast, the large neural network can keep extracting value from larger datasets, so its performance continues to rise as more labeled examples become available. That improvement, however, only happens if the team also has enough compute to train the bigger model on the larger dataset. The example shows that progress comes from both more data and the computational resources needed to use that data effectively, which is why the deep network benefits from scale while the simpler model stalls.

Key points:
- The simpler model levels off because it cannot use much more data effectively
- The neural network keeps improving as data increases
- Compute is needed to train the larger model on the larger dataset
- The example illustrates how data scale and compute together drive progress

Rubric:
Full credit identifies that the simpler model plateaus because of limited capacity, that the neural network can continue improving with more data, and that sufficient compute is required to realize that benefit. Partial credit for mentioning only the plateau or only the neural network's continued gains without linking both scale factors.

Explain why two models improve differently as a dataset expands.

Question:
A factory has only 900 labeled sensor records for detecting machine faults. Explain why carefully designed, expert-crafted features may work better than training a large neural network from scratch.

Sample answer:
With only 900 examples, a large neural network may not see enough data to learn reliable patterns and can overfit. Features designed from domain knowledge can inject useful structure into the model, which often improves performance when data is scarce.

Key points:
- Large neural networks usually benefit from more data
- Small datasets limit what a network can learn reliably
- Expert-crafted features add human knowledge
- This can be especially helpful in low-data settings

Rubric:
Full credit for explaining that limited data weakens the case for a large neural network and that expert-crafted features supply useful prior knowledge. Partial credit for mentioning only one of those ideas or giving an incomplete explanation.

Why can expert-crafted features help when labeled data is limited?

Which statement best describes why older deep learning ideas became influential later?

Two major forces behind recent deep learning gains

Recent progress in deep learning has mainly come from larger data sets and greater computing power.

Google

Machine learning is a branch of artificial intelligence focused on building systems that improve their task performance by learning from data or other experience. It draws on ideas from statistics, optimization, and data analysis to make predictions, recognize patterns, and support decisions.

Machine Learning

Representational learning is a class of machine learning that focuses on automatically discovering the most appropriate way to represent data. By learning these representations directly from the data, it eliminates the need for manual feature engineering and allows models to effectively process and understand complex inputs.

Representational Learning

Supervised learning is a method for learning a mapping from inputs x to outputs y from labeled examples (x, y). Common supervised learning algorithms include linear regression, logistic regression, and neural networks.

Supervised Learning

Kaggle is a popular online platform that hosts machine learning competitions. Each competition centers on a specific dataset, and many are sponsored by stakeholders who offer prizes for the winning solutions. The platform enables direct quantitative comparisons of competing approaches and fosters collaboration by allowing users to interact via forums and share code.

Kaggle Platform

Combining classical machine learning and deep learning within predictive analytics platforms enables users to forecast future trends and behaviors with high precision. These data-driven insights accelerate time-to-value across industries by optimizing courses of action and reducing margins of error, effectively replacing retrospective interpretations with forward-looking decision-making.

Predictive Analytics for Accelerated Decision-Making

A development set, sometimes called a holdout validation set, is a data split used to tune hyperparameters, compare feature choices, and make other model-design decisions during development.

Development Set

A test set is used to estimate how well a trained system performs, but it should not be used to choose the model, features, or hyperparameters.

Purpose of a Test Set

When a project must satisfy more than one evaluation criterion, a common strategy is to set one metric as a minimum acceptable requirement and then optimize a different metric among the solutions that meet that requirement. For example, a team might require inference time to stay under a limit and then select the model with the highest accuracy.

Combining Metrics With a Threshold and an Objective

For a brand-new machine learning task, it is usually very hard to know ahead of time which method will work best. Even skilled researchers often test many different ideas before they find one that performs well enough.

It Is Hard to Predict the Best Machine Learning Strategy Before Trying It

Machine learning development usually proceeds in cycles. A practical loop is to propose an approach, implement it, run an experiment to measure the result, and then use what you learned to choose the next change. Because each round reveals new information, teams often test many ideas before settling on one that works well.

An Iterative Machine Learning Workflow

In machine learning, two broad causes of poor performance are bias and variance. Separating them helps you choose the right next step, such as changing the model, gathering more data, or adjusting the training approach.

Bias and Variance as Two Major Sources of Error

With enough labeled input-output examples, an end-to-end model can learn to map inputs directly to outputs that are more complex than a single number. Those outputs can include text, images, audio, or other structured results.

What End-to-End Learning Can Predict

A pipeline can be poorly designed if the full system performs much worse than human level even though each stage is close to human level on the information it receives. In that case, the weakness is often that important information is being lost, so the design should be reconsidered.

When a Pipeline Is Missing Information

Text polarity detection is the task of deciding whether a piece of text expresses a favorable or unfavorable opinion, such as determining whether a product review praises or criticizes the item.

Text Polarity Detection

Machine learning strategy is the discipline of deciding which changes are most likely to improve a model, based on evidence from errors, data splits, and project constraints. It helps you prioritize the next experiment instead of guessing blindly.

Learn Before

Related

Learn After