Distributed representations can provide a statistical advantage when an apparently complicated structure can be compactly represented with a small number of parameters. In contrast, some traditional nondistributed learning algorithms generalize only due to the smoothness assumption, which states that if $$u \approx v$$, then the target function $$f$$ to be learned has the property that $$f(u) \approx f(v)$$ in general. While this assumption is useful, it suffers from the curse of dimensionality.

University of Michigan - Ann Arbor

Google

Distributed representations are representations composed of many elements that can be set separately from each other. They are powerful tools for representation learning because they can use $$n$$ features with $$k$$ values to describe $$k^n$$ different concepts. Since many deep learning algorithms are motivated by the assumption that hidden units can learn to represent the underlying causal factors that explain the data, distributed representations are naturally useful, as each direction in representation space can correspond to the value of a different underlying configuration variable.

Distributed Representations

The curse of dimensionality refers to the challenge where many machine learning problems become exceedingly difficult when the number of dimensions (features) in the data is high. Specifically, the number of possible configurations of variables increases exponentially with the number of dimensions. For a dataset with $$d$$ dimensions and $$v$$ values to be distinguished for each dimension, the space is divided into $$O(v^d)$$ regions, requiring an exponential number of examples to adequately cover all configurations.

The Curse of Dimensionality

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

If one associates certain pure symbols with a meaningful distributed representation, many of the things that can be said about one symbol can generalize to the other, and vice versa. Neural language models that operate on distributed representations of words generalize better than other models that operate directly on symbolic representations of words. Distributed representations induce a rich similarity space, in which semantically close concepts/inputs are close in distance, a property that is missing from purely symbolic representations.

Learn Before

Related