If one associates certain pure symbols with a meaningful distributed representation, many of the things that can be said about one symbol can generalize to the other, and vice versa. Neural language models that operate on distributed representations of words generalize better than other models that operate directly on symbolic representations of words. Distributed representations induce a rich similarity space, in which semantically close concepts/inputs are close in distance, a property that is missing from purely symbolic representations.

Generalization Due to Shared Attributes

Can have a statistical advantage when an apparently complicated structure can be compactly represented with a small number of parameters. Some traditional nondistributed learning algorithms generalize only due to the smoothness assumption, which states that if $u \approx v$, then the target function $f$ to be learned has the property that $f(u) \approx f(v)$ in general, which is useful, but suffers from the curse of dimensionality. 

Benefits of Distributed Representations

Representations composed of many elements that can be set separately from each other. They're powerful tools for representation learning, since they can use $n$ features with $k$ values to describe $k^n$ different concepts. Since many deep learning algorithms are motivated by the assumption that hidden units can learn to represent the underlying casual factors that explain the data, distributed representations are naturally useful, since each direction in representation space can correspond to the value of a different underlying configuration variable.

University of Michigan - Ann Arbor

- Feature learning, also called representation learning use machine learning to discover not only the mapping from representation to output but also the representation itself. A feature learning algorithm can discover a good set of features for a simple task in minutes, or for a complex task in hours to months. Feature learning can be either supervised or unsupervised.

- Representation of information to be processed has a great importance in day to day life since it eases the task of information processing. For example usage of Arabic numerals instead of Roman numerals for mathematical operations.

- In the context of machine learning an acceptable representation of data makes the future learning tasks easy. The successive learning tasks decide the choice of information representation.



Feature Learning (Representation Learning)

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Rules-based systems hard-code knowledge about the world in formal languages, which may lead to problems. System of rules can start of quite simple, but can become rather unwieldy over time as more and more exceptions and rule changes are added. What's more, when the data and scenarios change faster than you can update the rules, you can reach a point when you lose track of what is going on and how many exceptions there are. 

Different from rules-based systems, machine learning has the ability to acquire their own knowledge, by extracting patterns from raw data. Machine learning enabled computers to tackle problems involving knowledge of the real world and make decisions that appear subjective.

While classic machine learning only discovers the mapping from representation to output, representation learning discovers not only the mapping from representation to output but also the representation itself.  Learned representations often result in much better performance than can be obtained with hand-designed representations.

However, when high-level, abstract features such as a speaker's accent can be very difficult to be extracted from raw data, representation learning does not seem to help at first glance.  

Deep learning solves this central problem in representation learning by introducing representations that are expressed in terms of other, simpler representations. Deep learning enables the computer to build complex concepts out of simpler concepts.


Rules-Based Systems vs. Classic Machine Learning vs. Representation Learning vs. Deep Learning

Feature learning can be either supervised or unsupervised.

In supervised feature learning, features are learned using labeled input data. Examples include supervised neural networks, multilayer perceptron and (supervised) dictionary learning.

In unsupervised feature learning, features are learned with unlabeled input data. Examples include dictionary learning, independent component analysis, autoencoders, matrix factorization and various forms of clustering.


Methods of Feature Learning

Distributed Representations

Representations that may contain many entries but without significant meaningful separate control over each entry. Many learning algorithms are based on such representations, such as:
- Clustering methods
- K-nearest neighbors algorithms
- Decision trees
- Gaussian mixtures and mixtures of experts
- Kernel machines with a Gaussian (or other similarly local) kernel 
- Language or translation models based on n-grams

Nondistributed Representations

•	Pretraining encourages the learning algorithm to discover features that relate to the underlying causes that generate the observed data.
•	However, compared to other forms of unsupervised learning, unsupervised pretraining has the disadvantage of operating with two separate training phases.


How does Unsupervised Pretraining act as a regularizer?

•	In unsupervised pretraining, there is not a way of ﬂexibly adapting the strength of the regularization. In fact, we cannot adjust the strength of the regularization because there are many hyperparameters. The less hyperparameters there are, the easier it is to use them for prediction.
•	Another disadvantage of pretraining is that each phase has its own hyperparameters, and the second phase usually cannot be predicted by using the output of the first phase. Then, there is a delay between the two phases.



Disadvantage of Pretraining 

There are many instances where greedy unsupervised pretraining can be beneficial, but in other cases not so beneficial and can even be harmful. The most advantageous way to use greedy unsupervised pretraining is in the field of natural language processing, where one can pretrain once on a huge unlabeled set, learn a good representation, and then use the result for a supervised task. Greedy unsupervised pretraining is most successful when the number of unlabeled examples is either very large or very small.

When to use greedy unsupervised pretraining

- Greedy layer-wise unsupervised pretraining helped researchers  to train a supervised multi-layered deep neural network which did not incorporate complex deep neural network building blocks such  as convolution or recurrence.

- Each layer in a deep neural network is optimized in a greedy manner to obtain an optimized multi layered deep  neural network at the end.

- This type of pretraining requires a single-layer representation learning algorithms such as 
            - RBM( Restricted Boltzmann Machine)
            - a single layer autoencoder
            - a sparse coding model


Greedy Layer-Wise Unsupervised Pretraining

Data augmentation in data analysis are techniques used to increase the amount of data by adding slightly modified copies of already existing data or newly created synthetic data from existing data. 

Data Augmentation

Structured prediction or structured (output) learning is an umbrella term for supervised machine learning techniques that involves predicting structured objects, rather than scalar discrete or real values.

Learn Before

Related

Learn After