• Hidden layers predicts connection between inputs automatically, thats what deep learning is good at.

• Deep NN consists of more hidden layers (Deeper layers)

• Each Input will be connected to the hidden layer and the NN will decide the connections.

Simple NN and Deep NN

– For small data NN can perform as Linear regression or SVM (Support vector machine)

– For big data a small NN is better that SVM

– For big data a big NN is better that a medium NN is better that small NN.

– Hopefully we have a lot of data because the world is using the computer a little bit more

    - Mobiles

    - IOT (Internet of things)

Scale drives deep learning progress

• The main steps for building a Neural Network are:

     – Define the model structure (such as number of input features and outputs)

     – Initialize the model’s weights and biases.

     – Loop.

           ∗ Calculate current loss (forward propagation)

           ∗ Calculate current gradient (backward propagation)

           ∗ Update parameters (gradient descent)

• Preprocessing the dataset is important.

• Tuning the learning rate (which is an example of a “hyperparameter”) can make a big difference to the algorithm.

Main Step to build a NN

It is easy to build a perceptron that computes the logical functions OR and AND, however it is impossible to build one that computes logical XOR. This is because the perceptron is a linear classifier which acts as a decision boundary in two-dimensional space. 

Whilst OR and AND are linearly separable functions, XOR is not, it is impossible to draw a straight line that correctly separates its two categories. As shown in the figures below where $x_1$ and $x_2$ are inputs into the logical functions; filled circles represent outputs of 1, and white circles outputs of 0.

The XOR Problem

• Single neuron == linear regression without applying activation(perceptron)

• Basically a single neuron will calculate weighted sum of input(W.T*X) and then we can set a threshold to predict output in a perceptron. If weighted sum of input cross the threshold, perceptron fires and if not then perceptron doesn’t predict.

University of Michigan - Ann Arbor

https://www.coursera.org/learn/neural-networks-deep-learning?specialization=deep-learning

Neural Networks and Deep Learning

(1) Standard neural networks: real estate, online advertising

(2) Convolutional neural networks: applied in image recognition including photo tagging

(3) Recurrent Neural Networks: applied in sequence data including speech recognition, machine translation

(4) custom/hybrid neural networks: complex applications such as autonomous driving

application of neural networks in supervised learning

http://datahacker.rs/neural-networks-representation/

deep-learning representation

Because we have access to large columns of data, more computational power, and advanced algorithms.

For big data, large neural networks perform better than traditional ML models (SVM, logistic regression...)

Why is deep learning taking off?

(1) initialize parameters
(2) Implement the forward propagation to get the prediction
(3) compute the loss
(4) Implement the backward propagation to compute the derivatives of the parameters
(5) update the parameters
(6) repeat steps (2) to (5) to reduce the cost.

The learning circle of the neural network

- Start off trying to have one neural network do multiple things (can actually lead to better performance)
- differences and similarities amongst tasks is utilized to output a better result

Multi-task Learning

Benchmarking is a way to analyze how well your neural network is performing on a given standard set.
-Ensembling
-Multicrop at test time
-using open source code


Benchmarking

What is a Neural Network(NN)?

The working memory is the system that enables the ability to explicitly hold and manipulate pieces of information that are used to achieve a goal. Unlike humans, neural networks do not have a working memory.

Working Memory

Beyond standard categorical and numeric variables, deep learning practitioners frequently work with myriad unstructured data formats. These complex data types include text strings for natural language processing, images and point clouds for computer vision, and audio data. Handling these diverse formats requires specialized approaches and advanced tools compared to traditional tabular data.

Learn Before

Related

Learn After