Deep neural networks trained on data with mislabeled examples demonstrate a distinct learning trajectory: they initially fit the cleanly labeled examples and only later begin to interpolate the mislabeled ones. By terminating the training process after the clean data is learned but prior to the memorization of random labels, practitioners can effectively guarantee better generalization.

Learning Dynamics under Label Noise

In machine learning, interpolation of training data occurs when an over-parameterized model perfectly fits the training dataset, achieving zero training error. Because large neural networks possess vastly more parameters than necessary to fit the data, they have the capacity to memorize every training example. This tendency to interpolate is a key mechanism that allows deep neural networks to behave more like nonparametric models.

Claude

Interpolation is the process of estimating unknown values that fall between known values.

Interpolation

Dive into Deep Learning

This class of techniques is also sometimes referred to as Mixed Sample Data Augmentation (MSDA). In the nlp area, related work often interpolates inner components, more general mixing schemes, and add adversaries.

Learn Before

Related

Learn After