Rules-based systems hard-code knowledge about the world in formal languages, which may lead to problems. System of rules can start of quite simple, but can become rather unwieldy over time as more and more exceptions and rule changes are added. What's more, when the data and scenarios change faster than you can update the rules, you can reach a point when you lose track of what is going on and how many exceptions there are. 

Different from rules-based systems, machine learning has the ability to acquire their own knowledge, by extracting patterns from raw data. Machine learning enabled computers to tackle problems involving knowledge of the real world and make decisions that appear subjective.

While classic machine learning only discovers the mapping from representation to output, representation learning discovers not only the mapping from representation to output but also the representation itself.  Learned representations often result in much better performance than can be obtained with hand-designed representations.

However, when high-level, abstract features such as a speaker's accent can be very difficult to be extracted from raw data, representation learning does not seem to help at first glance.  

Deep learning solves this central problem in representation learning by introducing representations that are expressed in terms of other, simpler representations. Deep learning enables the computer to build complex concepts out of simpler concepts.


Rules-Based Systems vs. Classic Machine Learning vs. Representation Learning vs. Deep Learning

Feature learning can be either supervised or unsupervised.

In supervised feature learning, features are learned using labeled input data. Examples include supervised neural networks, multilayer perceptron and (supervised) dictionary learning.

In unsupervised feature learning, features are learned with unlabeled input data. Examples include dictionary learning, independent component analysis, autoencoders, matrix factorization and various forms of clustering.


Methods of Feature Learning

Representations composed of many elements that can be set separately from each other. They're powerful tools for representation learning, since they can use $n$ features with $k$ values to describe $k^n$ different concepts. Since many deep learning algorithms are motivated by the assumption that hidden units can learn to represent the underlying casual factors that explain the data, distributed representations are naturally useful, since each direction in representation space can correspond to the value of a different underlying configuration variable.

Distributed Representations

Representations that may contain many entries but without significant meaningful separate control over each entry. Many learning algorithms are based on such representations, such as:
- Clustering methods
- K-nearest neighbors algorithms
- Decision trees
- Gaussian mixtures and mixtures of experts
- Kernel machines with a Gaussian (or other similarly local) kernel 
- Language or translation models based on n-grams

Nondistributed Representations

•	Pretraining encourages the learning algorithm to discover features that relate to the underlying causes that generate the observed data.
•	However, compared to other forms of unsupervised learning, unsupervised pretraining has the disadvantage of operating with two separate training phases.


How does Unsupervised Pretraining act as a regularizer?

•	In unsupervised pretraining, there is not a way of ﬂexibly adapting the strength of the regularization. In fact, we cannot adjust the strength of the regularization because there are many hyperparameters. The less hyperparameters there are, the easier it is to use them for prediction.
•	Another disadvantage of pretraining is that each phase has its own hyperparameters, and the second phase usually cannot be predicted by using the output of the first phase. Then, there is a delay between the two phases.



Disadvantage of Pretraining 

There are many instances where greedy unsupervised pretraining can be beneficial, but in other cases not so beneficial and can even be harmful. The most advantageous way to use greedy unsupervised pretraining is in the field of natural language processing, where one can pretrain once on a huge unlabeled set, learn a good representation, and then use the result for a supervised task. Greedy unsupervised pretraining is most successful when the number of unlabeled examples is either very large or very small.

When to use greedy unsupervised pretraining

- Greedy layer-wise unsupervised pretraining helped researchers  to train a supervised multi-layered deep neural network which did not incorporate complex deep neural network building blocks such  as convolution or recurrence.

- Each layer in a deep neural network is optimized in a greedy manner to obtain an optimized multi layered deep  neural network at the end.

- This type of pretraining requires a single-layer representation learning algorithms such as 
            - RBM( Restricted Boltzmann Machine)
            - a single layer autoencoder
            - a sparse coding model


Greedy Layer-Wise Unsupervised Pretraining

Data augmentation in data analysis are techniques used to increase the amount of data by adding slightly modified copies of already existing data or newly created synthetic data from existing data. 

Data Augmentation

Structured prediction or structured (output) learning is an umbrella term for supervised machine learning techniques that involves predicting structured objects, rather than scalar discrete or real values.

Structured Prediction

- Feature learning, also called representation learning use machine learning to discover not only the mapping from representation to output but also the representation itself. A feature learning algorithm can discover a good set of features for a simple task in minutes, or for a complex task in hours to months. Feature learning can be either supervised or unsupervised.

- Representation of information to be processed has a great importance in day to day life since it eases the task of information processing. For example usage of Arabic numerals instead of Roman numerals for mathematical operations.

- In the context of machine learning an acceptable representation of data makes the future learning tasks easy. The successive learning tasks decide the choice of information representation.



University of Michigan - Ann Arbor

San Jose State University

There are several categories for ML algorithms, including:
 - Supervised Learning
 - Semi-supervised Learning
 - Unsupervised Learning
 - Reinforcement Learning
 - Feature Learning (Representation Learning)

The types of machine learning algorithms differ in their approach, the type of data they input and output, and the type of task or problem that they are intended to solve.


Machine Learning Categories

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Unsupervised machine learning methods find structure in unlabeled data.
They are used when we DON'T have prior knowledge about the categories.

Unsupervised statistical learning

In reinforcement learning, the machine goes through trial and error processes where it is rewarded/penalized for the actions it performs. The machine's goal is to maximize the total reward by leveraging previous attempts to make the next decision. The model starts from completely random trials and eventually leads to something with very sophisticated tactics and skills. Compared to supervised/unsupervised learning, reinforcement learning is much more advanced. A reinforcement model will continuously learn, unlike supervised/unsupervised models which all have an endpoint after the training and test data phases.

Reinforcement Learning

Feature Learning (Representation Learning)

https://www.amazon.com/Master-Algorithm-Ultimate-Learning-Machine-ebook/dp/B012271YB2

The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World

•	Inductive reasoning 
•	Connectionism
•	Evolutionary Computation 
•	Bayes’ theorem 
•	Analogical modeling 

Key takeaway: Pedro Domingos believes the “Master algorithm” will be a mixture of different schools of machine learning, as each school of thought has strengths and weaknesses that challenge their ability to be applied universally. 


Machine learning schools of thought (as explained in ”The Master Algorithm” by Pedro Domingos):

What are the categories of machine learning algorithms?

Supervised learning is a type of machine learning problem where the objective is to predict a designated unknown label based on known inputs. This approach requires a training dataset consisting of examples for which the ground truth labels are already known, allowing the model to learn the mapping from inputs to classifications.

Learn Before

Related

Learn After