Deep learning is a particular kind of machine learning that achieves great power and flexibility by representing the world as a nested hierarchy of concepts, with each concept defined in relation to simpler concepts, and more abstract representationss computed in terms of less abstract ones.


Deep Learning (in Machine learning)

Neural networks are made up of nodes (each representing a Logistic regression) with layers. They are trained to map an input vector X to an output vector Y based on the training data. The hidden layers of a network contain neurons, connections, and activation functions that occur between the input and output. An activation function is applied at each layer to correctly connect the input and output. Different types of networks may use diverse types of activation functions to correctly identify the nature of how the input and output are related.
In the figure out of the 4 neurons, the Logistic regression representation of two of them are shown.

Artificial Neural Networks Formulation

The goal of a neural network model is to approximate some function $f^*$. For example, for a classifier, $y = f^*(x)$ maps an input $x$ to a category $y$. A neural network model defines a mapping $y = f (x; \theta)$ and learns the values for the parameter $\theta$ that result in the best function approximation.

Artificial Neural Networks Objective

The McCulloch-Pitts model is an early, foundational computational model of artificial neurons developed by cyberneticists and neurophysiologists Warren McCulloch and Walter Pitts. This model predates modern deep learning but provided a natural starting point by abstracting the structure of biological neural systems into interconnected mathematical units.

McCulloch-Pitts Artificial Neuron Model

While early artificial neural networks drew heavily from biological nervous systems, modern deep learning is a broadly interdisciplinary field. Just as aeronautics innovation eventually moved beyond the direct study of birds, contemporary deep learning draws inspiration in equal or greater measure from mathematics, linguistics, psychology, statistics, and computer science, rather than relying exclusively on neuroscience.

Interdisciplinary Inspiration of Deep Learning

Artificial Neural Networks are built to resemble neurons in the human brain. The idea behind neural networks is to understand the pattern behind terms through examples rather than through task-specific rules.

University of Michigan - Ann Arbor

A network following the connectionist approach to cognitive modeling. Contains different kinds of units with connections between them. The units and connections are supposed to resemble neurons and their networks within the brain.

Connectionist network

Groups of neuron structures that work together in the brain. Supports the idea that the brain works like a symphony - different areas work together to create various cognitions.

Neural networks ( neuroscience )

- Generalized Additive Models
- Trees
- Random Forests
- Boosting
- Support Vector Machines
- Naive Bayes Classifier

Computer Intensive Classifiers

Receive signals from the hidden units and generate some output based on what is received.

Output units (connectionism)

Receives signals from the input units and sends them to the output units.

Hidden units (connectionism)

Activated by stimuli and send signals to the hidden units.

Input units (connectionism)

Determines the extent to which a signal increases or decreases activity of the unit it was sent to.

connection weight

Artificial Neural Networks

Example of activated connectionist network for living things and their properties

When the network generates wrong responses, an error signal is produced to help the learning process

Error signal

When a part of a connectionist network is damaged, the entire system does not fail, and performance is only affected incrementally. This is similar to the brain, since someone with damage may retain all cognitive functionality aside from the functions that damaged area is specialized to.

graceful degradation

The Random Forest is a model made up of many decision trees. It trains each one on a slightly different set of the observations, splitting nodes in each tree considering a limited number of the features. The final predictions of the random forest are made by averaging the predictions of each individual tree. In random forest algorithm, we do not need to do pruning for each "weak" decision tree. 

Advantage of Random Forest:
- Less overfitting
- Parallel implementation

Random Forest

Support Vector Machines are intended for binary classification (cases where there are two classes). The main goal is to find a hyperplane that best divides a dataset into two classes. This is quantified by achieving the maximum distance between data points of both classes separated by the hyperplane. The further away the data points are from the hyperplane, the more confidently we are able to classify future data points into their respective class.

Support Vector Machines

Boosting is another approach that can be used for both classification and regression. Boosting improves predictions of a decision tree by learning slowly. Unlike bagging, boosting works sequentially, where each tree grown uses the information from those created before it, and works on modified versions of the original data set. Boosting combines a large number of trees ($\hat{f}^{1}, ...,  \hat{f}^{B}$). Decision trees are fitted based on the residuals of the model rather than the response (Y), then update the residuals by adding it to the fitted function. This means that each model created is slowly reducing the error of the previous ones. This causes an improvement of $\hat{f}$, particularly in places where it does not do as well as others. Decision trees using boosting tend to be very small.


Boosting

A decision tree divides the prediction space into a series of simple regions according to the stratifying criteria. Decision trees can be applied to both regression and classification problems and can thus be of two types.

Decision Tree

Tree based methods aims to stratify or segment the predictor space (two predictors expressed in x y plane) into several simple regions. We can make a prediction for an observation by taking the mean or mode of the training observations in the region to which the observation belongs.

Tree based methods can be used both for 
-	Regression 
-	Classification


        


Tree-based Methods 

Generalized Additive Models (GAMs), similar to multiple linear regression models, attempt to predict a response value (Y) with many predictor variables ($X_1,...,X_p$). GAMs maintain their additivity by applying separate non-linear functions to predictor variables ($f_j(X_j)$) and adding them together. They can be further broken down into quantitative and qualitative responses.

Generalized Additive Models

Reference to Artificial Neural Networks

A method of supervised classification that takes many features, assumed to be independent, and uses Bayes' Theorem to determine given those independent variables how likely the likelihood of another event occurring.

It's called "Naive" because it assumes that features are conditionally independent, given the class, i.e., for all instances of a given class, the features have little or no correlation with each other.

Learn Before

Related

Learn After