A helpful website that introduces neural networks:
https://missinglink.ai/guides/neural-network-concepts/

Neural Network Reference

Convolutional networks, also known as convolutional neural networks, or CNNs, are a specialized kind of neural network for processing data that has a known grid-like topology. Examples include time-series data, which can be thought of as a 1-D grid taking samples at regular time intervals, and image data, which can be thought of as a 2-D grid of pixels. 

The network employs a mathematical operation called convolution. Convolution is a specialized kind of linear operation. Convolutional networks are simply neural networks that use convolution in place of general matrix multiplication in at least one of their layers.

The CNN gets its power from the following concepts:
- Sparse connectivity
- Parameter sharing

Convolutional Neural Network (CNN)

Recurrent neural networks, or RNNs, are a family of neural networks for processing sequential data. A recurrent neural network is specialized for processing a sequence of inputs $x^{(1)}, . . . , x^{(τ)}$ and each time add additional layers of comprehension on top of the previous inputs. 


Recurrent Neural Network (RNN)

**Generative Models**  As might be clear from their name , generative models are able to generate new contents by learning features of certain datasets. As for more formal definition, generative models specify in terms of a probabilistic model describing how it thinks the dataset was produced.  Using sampling from this probability distribution, we can create new data in the style of the old.
 
The joint probability distribution P(X,Y) is learned from the data, and then divided by P(X) to obtain the conditional probability distribution P(Y|X), which is used as a predictive model. It is called a generative model because the model represents the relationship between a given input X and an output Y. Common generative models are: Naive Bayes, Hidden Markov Model, Gaussian Mixture Model, Document Topic Generation Model (LDA), and restricted Boltzmann machine.

Generative Models

There are functions you can compute with a small layer that shallower networks would need a lot more hidden units in order to compute. 

Circuit Theory

https://www.simplilearn.com/deep-learning-algorithms-article

More about deep learning algorithms 

Data inputted into deep learning algorithms should be split into a distribution in which the majority of data (well over 70%+) is fed into the training set. This differs from traditional machine learning algorithms which had ~70% of the data given to the training set. This difference comes from the fact that there is now much more data available and deep learning algorithms improve with more data, whereas traditional machine learning algorithms improvement levels out after a certain amount of data is fed in.

For example, in a dataset of 1 million examples, you mighht decide that 10k examples are enough to evaluate which algorithm is better - 98% train, 1% dev, 1% test.

Deep learning train/dev/test split 

Deep feedforward networks, also called feedforward neural networks, or multilayer perceptrons (MLPs), are the quintessential deep learning models.

It was the first and simplest type of artificial neural network devised. In this network, the information moves in only one direction, forward, from the input nodes, through the hidden nodes (if any) and to the output nodes. There are no cycles or loops in the network.

Deep Feedforward Networks (MLP = Multi-Layer Perceptrons)

Instead of manually building up deep network architectures, many python libraries are available online to help users to easily implement deep learning models.

- Trax (built on TensorFlow)
- Tensor2Tensor (built on TensorFlow)
- Pytorch
- Fast.ai ( built on top of PyTorch)
- Keras (built to be a higher level framework to interface with other frameworks like TensorFlow)
- Vowpal Wabbit 
- Theano
- Caffee 

Deep Learning Python libraries (frameworks)

What can convolutional neural networks be used for?

A generative adversarial network (GAN) is a class of machine learning framework where two neural networks, a generative model and a discriminant model, contest against each other in a "game" (in the form of a zero-sum game, where one agent's gain is another agent's loss). Given a training set, this technique learns to generate new data with the same statistics as the training set. 

The core idea of a GAN is based on the "indirect" training through the discriminator, which itself is also being updated dynamically. This basically means that the generator is not trained to minimize the distance to a specific image, but rather to fool the discriminator and the discriminator is rewarded every time it correctly deduces if a sample is from the model distribution or the data distribution. This enables the model to learn in an unsupervised manner.

Generative adversarial network(GAN)

Deep Learning models require incredible computational resources to successfully train a model. As a result, optimizing the training process is a key step in developing an algorithm. One approach involves finding the parameters of a neural network that minimize its loss function. This is accomplished via gradient descent, a popular technique for minimizing loss function error. 

Optimization for Training Deep Models

Whereas deep learning algorithms were once always implemented on a single computer's central processing unit (CPU), research has shown that families of CPUs or even a single graphics processing unit (GPU) is much more effective. Some systems employ even greater parallelization to achieve the fastest training times with distributed architectures.

Implementations of Deep Learning

Monte Carlo algorithms return results with variability. The variability can typically be reduced by allocating more resources. The benefit of Monte Carlo algorithms is that they can provide an approximated estimation with limited source while maintaining a certain degree of accuracy. In Machine Learning, many problems are too complicated to have precise answers. Monte Carlo approximations are therefore a good approach to these problems.

Monte Carlo Methods

Deep learning frameworks provide specialized software tools that streamline the construction, training, and deployment of neural networks. The availability of these efficient frameworks has made the design and implementation of whole system optimization significantly easier, which is a critical component in achieving high performance for complex deep learning pipelines.

Deep Learning Frameworks

An adversarial example is an input to a machine learning model that has been intentionally modified with small, often imperceptible perturbations designed to cause the model to output an incorrect classification. For instance, adding specific, calculated noise to an image of a panda can cause a deep neural network to misclassify it as a gibbon, even though the image remains visually identical to the original for a human observer.

Adversarial Example

Deep learning algorithms can be implemented to solve real-world problems today.

Deep feedforward networks, also called multilayer perceptrons (MLPs), is the quintessential deep learning model that is behind nearly all modern practical applications of deep learning.

Scaling these models to large inputs such as high-resolution images or long temporal sequences requires specialization. 
 - Convolutional neural networks (CNN) can be used for processing large images.
 - Recurrent neural networks (RNN) can be used for processing temporal sequences. 


University of Michigan - Ann Arbor

Deep learning is a particular kind of machine learning that achieves great power and flexibility by representing the world as a nested hierarchy of concepts, with each concept defined in relation to simpler concepts, and more abstract representationss computed in terms of less abstract ones.


Deep Learning (in Machine learning)

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Deep Learning Algorithms

Rules-based systems hard-code knowledge about the world in formal languages, which may lead to problems. System of rules can start of quite simple, but can become rather unwieldy over time as more and more exceptions and rule changes are added. What's more, when the data and scenarios change faster than you can update the rules, you can reach a point when you lose track of what is going on and how many exceptions there are. 

Different from rules-based systems, machine learning has the ability to acquire their own knowledge, by extracting patterns from raw data. Machine learning enabled computers to tackle problems involving knowledge of the real world and make decisions that appear subjective.

While classic machine learning only discovers the mapping from representation to output, representation learning discovers not only the mapping from representation to output but also the representation itself.  Learned representations often result in much better performance than can be obtained with hand-designed representations.

However, when high-level, abstract features such as a speaker's accent can be very difficult to be extracted from raw data, representation learning does not seem to help at first glance.  

Deep learning solves this central problem in representation learning by introducing representations that are expressed in terms of other, simpler representations. Deep learning enables the computer to build complex concepts out of simpler concepts.


Rules-Based Systems vs. Classic Machine Learning vs. Representation Learning vs. Deep Learning

Deep learning analyses a training set, identifies complex patterns and applies them to new data. 

Reinforcement learning works sequentially in an unknown environment━taking an action, evaluating the rewards, and adjusting the following actions accordingly.

Deep learning and reinforcement learning complement each other:

 - Reinforcement learning algorithms manage the sequential process of taking an action, evaluating the result, and selecting the next best action. However, they need a good mechanism to select the best action based on previous interactions.
 - Deep learning can be that mechanism━it is the most powerful method available today to learn the best outcome based on previous data.

 

Deep Learning vs. Reinforcement Learning

Deep learning is taking off because as the amount of data and computational power increases, deep learning algorithms perform better than traditional machine learning methods. 

You can improve performance by increasing the scale of the NN or giving it more data. Increasing the scale of the NN often places greater strain on technological systems. The more accessible route is taking advantage of the increase in available data.

“Why is deep learning taking off?” 

(1) initialize parameters

(2) Implement the forward propagation to get the prediction 

(3) compute the loss 

(4) Implement the backward propagation to compute the derivatives of the parameters 

(5) update the parameters 

(6) repeat steps (2) to (5) to reduce the cost.

The learning circle of the neural network

1. Music Style Transfer
2. Voronoi Boundary Classification: A High-Dimensional Geometric Approach
via Weighted Monte Carlo Integration

Research Ideas for Deep Learning

- **High Avoidable bias**: if it does not fit training set well on cost function (e.g., comparable to human level performance), you might train a bigger network, train longer, switch to a better optimization algorithm, like the Adam, or change the neural network architecture/hyperparameter search.
- **High Variance**: if it does not fit dev set on cost function, you may use regularization, increase the size of training set, or change the neural network architecture/hyperparameter search.
- If it does not fit test set well on cost function, you may change or increase the size of dev set.
- If it does not perform well in real world (e.g., happy users), you may change or increase the size of test set or change the cost function.

Troubleshooting a deep learning model

(1) Standard neural networks: real estate, online advertising

(2) Convolutional neural networks: applied in image recognition including photo tagging

(3) Recurrent Neural Networks: applied in sequence data including speech recognition, machine translation

(4) custom/hybrid neural networks: complex applications such as autonomous driving

Applications of neural networks in supervised learning

Each datapoint: $(x, y)$   $X \in R^{n_{x} \times m}$   X.shape=$(n_{x}, m)$
$$X = \left[ \begin{array}{cccc}
    \vdots & \vdots & \cdots & \vdots \\
    X^{1} & X^{2} & \cdots & X^{m} \\
    \vdots & \vdots & \cdots & \vdots \\
\end{array} \right]$$
$Y = [y^{(1)}, y^{(2)}, ..., y^{(m)}]$   $Y \in R^{1 \times m}$   Y.shape=$(1, m)$
$m$ training examples: $[(x^{(1)}, y^{(1)}), (x^{(2)}, y^{(2)}), ..., (x^{(m)}, y^{(m)})]$
$m = m_{train}$   $m_{test}$ = # of test examples

Formulating the dataset in a Deep Learning Problem

A neural network configuration is called "shallow" if it has only a few hidden layers. In contrast, it's called "deep" if it has many hidden layers.

Deep vs. Shallow Neural Networks

Core concepts
-Machine learning
-Feature Engineering
-Feature Learning
-Deep Learning
Fundamental Concepts
-Logistic Regressino
-Artificial Neural Network
-Unit
-Artificial Neuron
-Activation Function
-Layer
Convolutional deep learning
-convolution
-pooling/subsampling
-convolutional neural network
-inception



Deep learning core concepts

The Universal Approximation Theorem (UAT) declares that only one hidden layer, with a finite number of neurons, is enough for approximating any looked-for function. This is an impressive statement for two reasons: On the one hand, this theorem proves the immense capacity of neural networks. But, on the other hand… Does it mean that we never need Deep Learning? No, breathe deeply, it doesn’t mean that…

The UAT doesn’t specify how many neurons it must contain. Although a single hidden layer could be enough to model a specific function, it could be more efficient to learn it by multiple hidden layers net. Furthermore, when training a net, we are looking for a function that best generalizes the relationship in data. Even if a single hidden network is able to represent the function that best fits the training examples, this would not mean that it generalizes better the behavior of the data out of the training set.

This is very well explained in the book Deep Learning by Ian Goodfellow, Yoshua Bengio, Aaron Courville:

In summary, a feedforward network with a single layer is sufficient to represent any function, but the layer may be infeasibly large and may fail to learn and generalize correctly. In many circumstances, using deeper models can reduce the number of units required to represent the desired function and can reduce the amount of generalization error.

When do we need Deep Learning?

Machine learning uses a set of algorithms to analyse and interpret data, learn from it, and based on the learnings, make best possible decisions. On the other hand, Deep learning structures the algorithms into multiple layers in order to create an “artificial neural network”. This neural network can learn from the data and make intelligent decisions on its own.


Machine Learning vs Deep Learning

Overfitting occurs when you achieve a good fit of your model on the training data, while it does not generalize well on new, unseen data. In other words, the model learned patterns specific to the training data, which are irrelevant in other data.
We can identify overfitting by looking at validation metrics, like loss or accuracy. Usually, the validation metric stops improving after a certain number of epochs and begins to decrease afterward. The training metric continues to improve because the model seeks to find the best fit for the training data.
There are several manners in which we can reduce overfitting in deep learning models. The best option is to get more training data. Unfortunately, in real-world situations, you often do not have this possibility due to time, budget or technical constraints.
Another way to reduce overfitting is to lower the capacity of the model to memorize the training data. As such, the model will need to focus on the relevant patterns in training data, which results in better generalization. In following children nodes, we’ll discuss three options to achieve this.

How to solve the overfitting problems in deep learning

Top 15 deep learning applications
1.self driving cars
2.healthcare (e.g. cancer detection)
3.Voice search and voice activated assistants
4.Automatically adding sounds to silent movies
5.Automatic machine translation
6.Automatic text generation
7.Automatic handwriting generation
8.Image recognition
9.automatic image caption generation
10.automatic colorization
11.Advertising
12.Predicting earthquakes
13.Neural networks for brain cancer detection
14.Neural networks in finance
15.Energy market price forecasting

Top 15 deep learning applications

Deep learning has a rich history that comes under different names. It develops with accelerated speed due to the increasing size of datasets and capacity of computer infrastructures in the recent decades. The complexity of tasks that deep learning can handle has increased.

Deep Learning History

- Deep Learning
- Deep learning and application in cancer detection
- Deep learning in a nutshell
- Top 15 Deep Learning applications that will rule the world in 2018 and beyond
- Delve into ReLU
- The ABCs of Approximate Bayes Computation
- Convolutional Recurrent Neural Networks For Music Classification.

Deep Learning (in Machine Learning) References

The inability for traditional algorithms to generalize complex problems was what inspired machine learning in the first place. Today, a similar situation has arose that complex problems with high dimensional inputs (high number of features) are not generalizable with most simple machine learning algorithms, not to mention require high computational power to solve. This is what drives the development of deep learning, as it will solve some problems better than machine learning can.

Challenges Motivating Deep Learning

FAKE images and videos including facial information generated by digital manipulation, in particular with DeepFake methods, have become a great public concern recently. The very popular term “DeepFake” is referred to a deep-learning based technique able to create fake videos by swapping the face of a person by the face of another person.

DeepFake

This paper is based on the working of a transformer model. A transformer model basically helps in transforming a sequence of input into another depending on the problem statement.
These include translation of a language to another, or an answer for a question, with the help of an encoder and decoder model stacked together.

Attention is all you Need (Presentation)

To effectively teach a complex, computer-related topic, especially to a young audience like a 10-year-old, it is crucial to use simple examples. This approach helps make abstract ideas more concrete and understandable. The explanation should ideally come from an expert with deep knowledge of the subject, such as a computer scientist explaining deep learning.

Explaining Complex Concepts with Simple Examples

A machine learning system is being designed to identify different species of birds in photographs. The model first learns to recognize basic elements like lines, curves, and color gradients. In subsequent stages, it combines these basic elements to identify more complex components like feathers, beaks, and eyes. Finally, it uses the arrangement of these components to classify the bird species. Which statement best analyzes the fundamental principle that gives this approach its power?

A data science team is deciding between two projects. Evaluate which project is a more suitable candidate for a machine learning approach that builds complex concepts out of simpler ones, and justify your decision by explaining how the core principle of this approach applies to the problem's complexity and data structure.

Choosing the Right Machine Learning Approach

A machine learning model is tasked with identifying a cat in an image. Arrange the following stages of representation in the order they would likely be learned by a system that builds complex concepts from simpler ones, starting from the most basic input.

End-to-end training is a core methodology in deep learning where a complete system is built and tuned jointly to map raw inputs to final outputs. This approach contrasts with traditional machine learning pipelines that assemble individually optimized components, such as a separate, manually engineered feature extractor followed by a shallow classifier. By simultaneously training all layers of computation, the model automatically discovers the optimal representations needed for the task.

Learn Before

Related

Learn After