While deep neural networks have the capacity to fit arbitrary, random, or incorrect labels, this capability typically only emerges after many training iterations. In the presence of label noise, neural networks demonstrate a distinct learning trajectory: they initially fit the cleanly labeled examples and only subsequently begin to interpolate the mislabeled data. Because of this dynamic, terminating the training process after the clean data is learned—but before the random labels are memorized—provides a strong guarantee of better generalization.