While the XOR function cannot be calculated by a single perceptron, it can be calculated by a layered network of units; for example, using two layers of ReLU-based units. As shown in the figure below, where the numbers on the arrows represent the weights for each unit and $h_1$ and $h_2$ form a hidden layer, which represents the input in a new linearly separable $h$ space.

Note that the solution to the XOR problem requires a network of units with non-linear activation functions. This is because a network formed by many layers of purely linear units can always be reduced a single layer of linear units with appropriate weights.

Solution to the XOR Problem: Neural Networks

A single perceptron can easily compute linearly separable logical functions like OR and AND, but it cannot compute the logical XOR function. Because a perceptron acts as a linear classifier, it creates a straight decision boundary in a two-dimensional space. While the outputs of OR and AND can be separated by a straight line, the outputs of XOR cannot be linearly separated. As shown in the accompanying figure, where $$x_1$$ and $$x_2$$ are inputs, no straight line can correctly separate the filled circles (outputs of 1) from the white circles (outputs of 0).

University of Michigan - Ann Arbor

Google

• Single neuron == linear regression without applying activation(perceptron)

• Basically a single neuron will calculate weighted sum of input(W.T*X) and then we can set a threshold to predict output in a perceptron. If weighted sum of input cross the threshold, perceptron fires and if not then perceptron doesn’t predict.

What is a Neural Network(NN)?

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

• Hidden layers predicts connection between inputs automatically, thats what deep learning is good at.

• Deep NN consists of more hidden layers (Deeper layers)

• Each Input will be connected to the hidden layer and the NN will decide the connections.

Simple NN and Deep NN

– For small data NN can perform as Linear regression or SVM (Support vector machine)

– For big data a small NN is better that SVM

– For big data a big NN is better that a medium NN is better that small NN.

– Hopefully we have a lot of data because the world is using the computer a little bit more

    - Mobiles

    - IOT (Internet of things)

Scale drives deep learning progress

• The main steps for building a Neural Network are:

     – Define the model structure (such as number of input features and outputs)

     – Initialize the model’s weights and biases.

     – Loop.

           ∗ Calculate current loss (forward propagation)

           ∗ Calculate current gradient (backward propagation)

           ∗ Update parameters (gradient descent)

• Preprocessing the dataset is important.

• Tuning the learning rate (which is an example of a “hyperparameter”) can make a big difference to the algorithm.

Learn Before

Related

Learn After