Can have a statistical advantage when an apparently complicated structure can be compactly represented with a small number of parameters. Some traditional nondistributed learning algorithms generalize only due to the smoothness assumption, which states that if $u \approx v$, then the target function $f$ to be learned has the property that $f(u) \approx f(v)$ in general, which is useful, but suffers from the curse of dimensionality. 

University of Michigan - Ann Arbor

Representations composed of many elements that can be set separately from each other. They're powerful tools for representation learning, since they can use $n$ features with $k$ values to describe $k^n$ different concepts. Since many deep learning algorithms are motivated by the assumption that hidden units can learn to represent the underlying casual factors that explain the data, distributed representations are naturally useful, since each direction in representation space can correspond to the value of a different underlying configuration variable.

Distributed Representations

The curse of dimensionality states that as the number of dimensions (features) in the data increase, the number of categories of interest increases exponentially. This makes using machine learning on high dimensional data difficult.

For a data set with $d$ dimensions and $v$ values to be categorized for each dimension, $O(v^d)$ regions and examples are needed.


The Curse of Dimensionality

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

If one associates certain pure symbols with a meaningful distributed representation, many of the things that can be said about one symbol can generalize to the other, and vice versa. Neural language models that operate on distributed representations of words generalize better than other models that operate directly on symbolic representations of words. Distributed representations induce a rich similarity space, in which semantically close concepts/inputs are close in distance, a property that is missing from purely symbolic representations.

Learn Before

Related