While the XOR function cannot be calculated by a single perceptron, it can be calculated by a layered network of units; for example, using two layers of ReLU-based units. As shown in the figure below, where the numbers on the arrows represent the weights for each unit and $h_1$ and $h_2$ form a hidden layer, which represents the input in a new linearly separable $h$ space.

Note that the solution to the XOR problem requires a network of units with non-linear activation functions. This is because a network formed by many layers of purely linear units can always be reduced a single layer of linear units with appropriate weights.

Solution to the XOR Problem: Neural Networks

It is easy to build a perceptron that computes the logical functions OR and AND, however it is impossible to build one that computes logical XOR. This is because the perceptron is a linear classifier which acts as a decision boundary in two-dimensional space. 

Whilst OR and AND are linearly separable functions, XOR is not, it is impossible to draw a straight line that correctly separates its two categories. As shown in the figures below where $x_1$ and $x_2$ are inputs into the logical functions; filled circles represent outputs of 1, and white circles outputs of 0.

University of Michigan - Ann Arbor

• Single neuron == linear regression without applying activation(perceptron)

• Basically a single neuron will calculate weighted sum of input(W.T*X) and then we can set a threshold to predict output in a perceptron. If weighted sum of input cross the threshold, perceptron fires and if not then perceptron doesn’t predict.

What is a Neural Network(NN)?

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

• Hidden layers predicts connection between inputs automatically, thats what deep learning is good at.

• Deep NN consists of more hidden layers (Deeper layers)

• Each Input will be connected to the hidden layer and the NN will decide the connections.

Simple NN and Deep NN

– For small data NN can perform as Linear regression or SVM (Support vector machine)

– For big data a small NN is better that SVM

– For big data a big NN is better that a medium NN is better that small NN.

– Hopefully we have a lot of data because the world is using the computer a little bit more

    - Mobiles

    - IOT (Internet of things)

Scale drives deep learning progress

• The main steps for building a Neural Network are:

     – Define the model structure (such as number of input features and outputs)

     – Initialize the model’s weights and biases.

     – Loop.

           ∗ Calculate current loss (forward propagation)

           ∗ Calculate current gradient (backward propagation)

           ∗ Update parameters (gradient descent)

• Preprocessing the dataset is important.

• Tuning the learning rate (which is an example of a “hyperparameter”) can make a big difference to the algorithm.

Learn Before

Related

Learn After