Although deep neural networks possess millions of parameters that are updated and saved during training, their behavior can often be more fruitfully understood as nonparametric. Because neural networks are massively over-parametrized—having many more parameters than are needed to fit the training data—they tend to interpolate the training data, achieving zero training error. This capacity to perfectly memorize the dataset mirrors the behavior of nonparametric models, indicating that deep networks exhibit a level of complexity that scales similarly to nonparametric methods.