If one associates certain pure symbols with a meaningful distributed representation, many of the things that can be said about one symbol can generalize to the other, and vice versa. Neural language models that operate on distributed representations of words generalize better than other models that operate directly on symbolic representations of words. Distributed representations induce a rich similarity space, in which semantically close concepts/inputs are close in distance, a property that is missing from purely symbolic representations.

University of Michigan - Ann Arbor

Representations composed of many elements that can be set separately from each other. They're powerful tools for representation learning, since they can use $n$ features with $k$ values to describe $k^n$ different concepts. Since many deep learning algorithms are motivated by the assumption that hidden units can learn to represent the underlying casual factors that explain the data, distributed representations are naturally useful, since each direction in representation space can correspond to the value of a different underlying configuration variable.

Distributed Representations

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Generalization Due to Shared Attributes

Can have a statistical advantage when an apparently complicated structure can be compactly represented with a small number of parameters. Some traditional nondistributed learning algorithms generalize only due to the smoothness assumption, which states that if $u \approx v$, then the target function $f$ to be learned has the property that $f(u) \approx f(v)$ in general, which is useful, but suffers from the curse of dimensionality. 

Learn Before

Related