For a feedforward neural network, the depth of the network is the number of hidden layers plus one (as the output layer is also parameterized). The width of the network is the dimensionality of its hidden layers.

The main architectural considerations for designing a neural network are choosing the depth of the network and the width of each layer.


Depth and Width for Neural Networks

In the chain structure example $f (x) = f ^{(3)}(f^{(2)}(f^{(1)}(x)))$, $f^{(1)}$ is called
the first layer of the network, f^{(2)} is called the second layer, and so on. The final layer of a feedforward network is called the output layer. 

During neural network training, we drive $f(x)$ to match $f^∗(x)$. The training data provides us with noisy, approximate examples of $f^∗(x)$ evaluated at different training points. Each example x is accompanied by a label $y ≈ f^*(x)$.

The training examples specify directly that the output layer must produce a value that is close to $y$ at each point $x$. However, the behavior of the other layers is not directly specified by the training data. The training data do not say what each individual layer should do. Instead, the learning algorithm must decide how to use these layers to best implement an approximation of $f^*$. Because the training data does not show the desired output for each of these layers, they are called hidden layers.

Layers of a Feed Forward Neural Network

$L$ = # of layers
$m$ = # of training datapoints
$$n^{[l]} = $$ # of units (neurons) in layer $l$
$$A^{[l]} = g^{[l]}(Z^{[l]}) = $$ activations (outputs) in layer $l$
$Z^{[l]}, A^{[l]}: (n^{[l]},m)$
$$X = A^{[0]}$$      $$\hat{Y} = A^{[L]}$$
$$W^{[l]} = $$ Weights for $$Z^{[l]}$$      $$W^{[l]}: (n^{[l]}, n^{[l - 1]})$$
$$b^{[l]} = $$ Biases for $$Z^{[l]}$$      $$b^{[l]}: (n^{[l]}, 1)$$

Formulation of a Feedforward Neural Networks

The universal approximation theorem (Horniket al., 1989; Cybenko, 1989) states that a feedforward network with a linear output layer and at least one hidden layer with any “squashing” activation function (such as the logistic sigmoid activation function) can approximate any Borel measurable function from one ﬁnite-dimensional space to another with any desired nonzero amount of error, provided that the network is given enough hidden units. The derivatives of the feedforward network can also approximate the derivatives of the function arbitrarily well.

It means that regardless of what function we are trying to learn, we know that a large feedforward network will be able to represent this function. 

Universal Approximation Theorem

Feedforward neural networks are called networks because they are typically represented by composing together many different functions. The model is associated with a directed acyclic graph describing how the functions are composed together.

For example, we might have three functions $f^{(1)} , f ^{(2)}$, and $f^{(3)}$ connected in a chain, to form $f (x) = f ^{(3)}(f^{(2)}(f^{(1)}(x)))$. These chain structures are the most commonly used structures of neural networks. 


University of Michigan - Ann Arbor

Deep feedforward networks, also called feedforward neural networks, or multilayer perceptrons (MLPs), are the quintessential deep learning models.

It was the first and simplest type of artificial neural network devised. In this network, the information moves in only one direction, forward, from the input nodes, through the hidden nodes (if any) and to the output nodes. There are no cycles or loops in the network.

Deep Feedforward Networks (MLP = Multi-Layer Perceptrons)

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Overall Structure of Deep Feedforward Networks

A Feed forward neural network model training occurs in six stages:

- Iterate until convergence
   - Initialization
   - Forward propagation
   - Error function (Objective function)
   - Backpropagation
   - Weight update

At the end of this process, the model is ready to make predictions for unknown input data. New data can be fed to the model, a forward pass is performed, and the model generates its prediction.

Stages of Feed Forward Neural Network Learning

A hyperparameter is a setting that affects the structure or operation of the neural network. In real deep learning projects, tuning hyperparameters is the primary way to build a network that provides accurate predictions for a certain problem. 

Hyperparameters of Feedforward Neural Network

There are a few different types of cost functions, but for the most part the same type of cost function is used for neural networks. We usually use the principle of maximum likelihood for most cases, such as when our parametric model uses a distribution p(y | x;θ), where the relationship between the input (training data) and output (model's predictions) of the model as the cost function. A sometimes simpler approach used, where we are predicting some statistic of y conditioned on x, instead of predicting a complete probability distribution over y. 

Deep Feedforward Network Cost Functions

-The chain rule that underlies the back-propagation algorithm (one of the common algorithms used to train neural networks) was invented in the seventeenth century.

-While algebra and advanced mathematics have always been implemented to solve computer optimization problems, gradient descent was not introduced as a technique for iteratively approximating the solution to optimization problems until the nineteenth century.

-Non-linear functions could only be properly studied once a multilayer perceptron and a means of computing the gradient were developed. One of the biggest advancements for this development occurred once eﬃcient applications of the chain rule based on dynamic programming were implemented between the 1960s and 70s. 


Deep Feedfoward Network as time passed.

Approaches to back-propagation taking a computational graph and a set ofnumerical values for the inputs to the graph and returning a set of numerical valuesdescribing the gradient at those input values is called symbol-to-number diﬀerentiation.

Symbol-to-number Diﬀerentiation

Theano and Tensorflow Approach is to take a computational graph and add additional nodes to the graph that provide a symbolic description of the desired derivatives.

Theano and Tensorflow Approach

These algebraic and graph-based representations operating on symbols, orvariables that do not have speciﬁc values are called symbolic representations.

Symbolic Representations

Language modeling is one of the application of feedforward networks: predicting upcoming words from prior word context. Neural language modeling
is an important NLP task in itself, and it plays a role in many important algorithms for tasks like machine translation, summarization, speech recognition, grammar correction, and dialogue.



Feedforward Neural Language Modeling

Most Neural NLP classifications take word embeddings as their inputs.
$$ x = [e_{w1};e_{w2};e_{w3};...;e_{wn}]$$
These kind of embeddings are gained from the pertaining process relying on another algorithm.
Embedding can also be learned as part of the NLP task in the neural language modeling task.



Learn Before

Related

Learn After