Deep learning is a particular kind of machine learning that achieves great power and flexibility by representing the world as a nested hierarchy of concepts, with each concept defined in relation to simpler concepts, and more abstract representationss computed in terms of less abstract ones.


Deep Learning (in Machine learning)

A neural network maps an input vector $$X$$ to an output vector $$Y$$ by stacking layers of nodes, each functioning as a logistic-regression unit: a weighted sum of inputs passed through an activation function. The hidden layers contain neurons, the connections between them, and the activation functions applied at each layer to transform the input into the output. The accompanying figure shows the logistic-regression representation of two of the network's four neurons.

Artificial Neural Networks Formulation

The goal of a neural network model is to approximate some function $$f^*$$. For example, for a classifier, $$y = f^*(x)$$ maps an input $$x$$ to a category $$y$$. A neural network model defines a mapping $$y = f(x; \theta)$$ and learns the values for the parameter $$\theta$$ that result in the best function approximation.

Artificial Neural Networks Objective

The McCulloch-Pitts model is an early, foundational computational model of artificial neurons developed by cyberneticists and neurophysiologists Warren McCulloch and Walter Pitts. This model predates modern deep learning but provided a natural starting point by abstracting the structure of biological neural systems into interconnected mathematical units.

McCulloch-Pitts Artificial Neuron Model

While early artificial neural networks drew heavily from biological nervous systems, modern deep learning is a broadly interdisciplinary field. Just as aeronautics innovation eventually moved beyond the direct study of birds, contemporary deep learning draws inspiration in equal or greater measure from mathematics, linguistics, psychology, statistics, and computer science, rather than relying exclusively on neuroscience.

Interdisciplinary Inspiration of Deep Learning

An Artificial Neural Network (ANN) is a computational model inspired by the structure of biological neural networks in the brain, made up of interconnected neuron-like units organized in layers. Rather than following task-specific rules, an ANN learns to recognize patterns in data from examples, adjusting the connections between its units during training.

University of Michigan - Ann Arbor

Google

A network following the connectionist approach to cognitive modeling. Contains different kinds of units with connections between them. The units and connections are supposed to resemble neurons and their networks within the brain.

Connectionist network

Groups of neuron structures that work together in the brain. Supports the idea that the brain works like a symphony - different areas work together to create various cognitions.

Neural networks ( neuroscience )

- Generalized Additive Models
- Trees
- Random Forests
- Boosting
- Support Vector Machines
- Naive Bayes Classifier

Computer Intensive Classifiers

Receive signals from the hidden units and generate some output based on what is received.

Output units (connectionism)

Receives signals from the input units and sends them to the output units.

Hidden units (connectionism)

Activated by stimuli and send signals to the hidden units.

Input units (connectionism)

Determines the extent to which a signal increases or decreases activity of the unit it was sent to.

connection weight

Artificial Neural Networks

Example of activated connectionist network for living things and their properties

When the network generates wrong responses, an error signal is produced to help the learning process

Error signal

When a part of a connectionist network is damaged, the entire system does not fail, and performance is only affected incrementally. This is similar to the brain, since someone with damage may retain all cognitive functionality aside from the functions that damaged area is specialized to.

graceful degradation

The Random Forest is a model made up of many decision trees. It trains each one on a slightly different set of the observations, splitting nodes in each tree considering a limited number of the features. The final predictions of the random forest are made by averaging the predictions of each individual tree. In random forest algorithm, we do not need to do pruning for each "weak" decision tree. 

Advantage of Random Forest:
- Less overfitting
- Parallel implementation

Random Forest

Support Vector Machines are intended for binary classification (cases where there are two classes). The main goal is to find a hyperplane that best divides a dataset into two classes. This is quantified by achieving the maximum distance between data points of both classes separated by the hyperplane. The further away the data points are from the hyperplane, the more confidently we are able to classify future data points into their respective class.

Support Vector Machines

Boosting is another approach that can be used for both classification and regression. Boosting improves predictions of a decision tree by learning slowly. Unlike bagging, boosting works sequentially, where each tree grown uses the information from those created before it, and works on modified versions of the original data set. Boosting combines a large number of trees ($\hat{f}^{1}, ...,  \hat{f}^{B}$). Decision trees are fitted based on the residuals of the model rather than the response (Y), then update the residuals by adding it to the fitted function. This means that each model created is slowly reducing the error of the previous ones. This causes an improvement of $\hat{f}$, particularly in places where it does not do as well as others. Decision trees using boosting tend to be very small.


Boosting

A decision tree divides the prediction space into a series of simple regions according to the stratifying criteria. Decision trees can be applied to both regression and classification problems and can thus be of two types.

Decision Tree

Tree based methods aims to stratify or segment the predictor space (two predictors expressed in x y plane) into several simple regions. We can make a prediction for an observation by taking the mean or mode of the training observations in the region to which the observation belongs.

Tree based methods can be used both for 
-	Regression 
-	Classification


        


Tree-based Methods 

Reference to Artificial Neural Networks

A method of supervised classification that takes many features, assumed to be independent, and uses Bayes' Theorem to determine given those independent variables how likely the likelihood of another event occurring.

It's called "Naive" because it assumes that features are conditionally independent, given the class, i.e., for all instances of a given class, the features have little or no correlation with each other.

(Naive) Bayes Classifier

Generalized Additive Models (GAMs), similar to multiple linear regression models, attempt to predict a response value ($$Y$$) using many predictor variables ($$X_1, \dots, X_p$$). GAMs maintain their additivity by applying separate non-linear functions to each predictor variable ($$f_j(X_j)$$) and adding them together. They can be applied to both quantitative and qualitative responses.

Learn Before

Related

Learn After