A helpful website for understanding how CNN works:
https://towardsdatascience.com/a-comprehensive-guide-to-convolutional-neural-networks-the-eli5-way-3bd2b1164a53

CNN Reference

CNNs are currently being applied to several applications, including:

 - Computer vision—medical image analysis, image recognition, face detection and video classification
 - Natural Language Processing (NLP)—semantic parsing, sentence modeling and search query retrieval
 - Drug discovery—the discovery of chemical features and prediction of medicinal benefits

Applications of Convolutional Neural Networks

CNNs and RNNs are not mutually exclusive, as both can perform classification of image and text inputs, creating an opportunity to combine the two network types for increased effectiveness. This is especially true if the input to be classified is visually complex with added temporal characteristics that a CNN alone would be unable to process.

Typically, when these two network types are combined, sometimes referred to as a CRNN, inputs are first processed by CNN layers whose outputs are then fed to RNN layers. CNN Long Short-Term Memory (LSTM) architectures are particularly promising, as they facilitate analysis of inputs over longer periods than could be achieved with lower-level RNN architecture types.

Currently, these hybrid architectures are being explored for use in applications like video scene labeling, emotion detection or gesture recognition, video identification or gait recognition, and DNA sequence prediction.

Hybrid of Convolutional and Recurrent Neural Network

- Python: conv_forward
- TensorFlow: tf.nn.conv2d
- Keras: Conv2D
- Pytorch: torch.nn.Conv2d
- Caffe: message ConvolutionParameter
- Theano: theano.tensor.nnet.conv.conv2d

Methods to Calculate Convolution in Python

CNNs are made up of three-layer types—convolutional, pooling and fully-connected (FC).

In the convolutional layers, the input matrix is passed through a set of filters that output a feature map. This output is then sent to a pooling layer, which reduces the size of the feature map. This helps reduce the processing time by condensing the map to its most essential information.

The convolutional and pooling processes are repeated several times, with the number of repeats depending on the network, after which the condensed feature map outputs are sent to a series of FC layers. These FC layers then flatten the maps together and compare the probabilities of each feature occurring in conjunction with the others, until the best classification is determined.

Convolutional Neural Networks Architecture

A 3D CNN is similar to a 2D CNN except that it differs in (non-exhaustive listing):
1- 3D Convolution Layers

2- 3D MaxPool Layers

3D Convolutional Neural Network

Visualizing and Understanding Convolutional Networks Paper

- Convolutional neural network can be used for obtaining high- dimensional images rather than being used in class prediction for logistic regression  or value prediction for regression problem

- Output of the CNN model can be a tensor S(i,j,k) which represent the pixel(j,k) of ith class, such a collaborative  effort can be used to precisely form a perfect image as the output.

Strategies to get output dimension same as the input are 

- Pooling is completely avoided or can use pooling with unit stride.

- Pixel-wise labelling involves initial guess of image pixels and refining this guess by interacting with neighboring pixels by parameter sharing.

Structured Output from CNN

A Convolutional Recurrent Neural Network (CRNN) is a combination of a Convolutional Neural Network (CNN) and a Recurrent Neural Network (RNN). 
-CNN are specialized networks for processing data that use a specialized, linear mathematical operation, also known as a convolution. 
-RNN is specialized for processing a sequence of inputs $x^{(1)}, . . . , x^{(τ)}$ and each time adds additional layers of comprehension on top of the previous inputs. 
-The goal of a CRNN is to take advantage of CNNs for local feature extraction and RNNs for temporal summarization of the extracted features.


Convolutional Recurrent Neural Network (CRNN) 

Questions about the ReLU.

Convolutional networks, also known as convolutional neural networks, or CNNs, are a specialized kind of neural network for processing data that has a known grid-like topology. Examples include time-series data, which can be thought of as a 1-D grid taking samples at regular time intervals, and image data, which can be thought of as a 2-D grid of pixels. 

The network employs a mathematical operation called convolution. Convolution is a specialized kind of linear operation. Convolutional networks are simply neural networks that use convolution in place of general matrix multiplication in at least one of their layers.

The CNN gets its power from the following concepts:
- Sparse connectivity
- Parameter sharing

University of Michigan - Ann Arbor

Deep learning algorithms can be implemented to solve real-world problems today.

Deep feedforward networks, also called multilayer perceptrons (MLPs), is the quintessential deep learning model that is behind nearly all modern practical applications of deep learning.

Scaling these models to large inputs such as high-resolution images or long temporal sequences requires specialization. 
 - Convolutional neural networks (CNN) can be used for processing large images.
 - Recurrent neural networks (RNN) can be used for processing temporal sequences. 


Deep Learning Algorithms

Neural relation extraction (NRE) models introduce neural networks to automatically extract semantic features from text. Compared with SRE models,
NRE methods can effectively capture textual information and generalize to wider range of data.

Neural Relation Extraction Models

Various architecture models are used to capture the interaction between entities and relations.

 - Linear/Bilinear Models
 - Factorization Models
 - Neural Networks
 - Convolutional Neural Networks
 - Recurrent Neural Networks
 - Transformers
 - Graph Neural Networks (GNNs)


Encoding Models

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

A helpful website that introduces neural networks:
https://missinglink.ai/guides/neural-network-concepts/

Neural Network Reference

Convolutional Neural Network (CNN)

Recurrent neural networks, or RNNs, are a family of neural networks for processing sequential data. A recurrent neural network is specialized for processing a sequence of inputs $x^{(1)}, . . . , x^{(τ)}$ and each time add additional layers of comprehension on top of the previous inputs. 


Recurrent Neural Network (RNN)

**Generative Models**  As might be clear from their name , generative models are able to generate new contents by learning features of certain datasets. As for more formal definition, generative models specify in terms of a probabilistic model describing how it thinks the dataset was produced.  Using sampling from this probability distribution, we can create new data in the style of the old.
 
The joint probability distribution P(X,Y) is learned from the data, and then divided by P(X) to obtain the conditional probability distribution P(Y|X), which is used as a predictive model. It is called a generative model because the model represents the relationship between a given input X and an output Y. Common generative models are: Naive Bayes, Hidden Markov Model, Gaussian Mixture Model, Document Topic Generation Model (LDA), and restricted Boltzmann machine.

Generative Models

There are functions you can compute with a small layer that shallower networks would need a lot more hidden units in order to compute. 

Circuit Theory

https://www.simplilearn.com/deep-learning-algorithms-article

More about deep learning algorithms 

Data inputted into deep learning algorithms should be split into a distribution in which the majority of data (well over 70%+) is fed into the training set. This differs from traditional machine learning algorithms which had ~70% of the data given to the training set. This difference comes from the fact that there is now much more data available and deep learning algorithms improve with more data, whereas traditional machine learning algorithms improvement levels out after a certain amount of data is fed in.

For example, in a dataset of 1 million examples, you mighht decide that 10k examples are enough to evaluate which algorithm is better - 98% train, 1% dev, 1% test.

Deep learning train/dev/test split 

Deep feedforward networks, also called feedforward neural networks, or multilayer perceptrons (MLPs), are the quintessential deep learning models.

It was the first and simplest type of artificial neural network devised. In this network, the information moves in only one direction, forward, from the input nodes, through the hidden nodes (if any) and to the output nodes. There are no cycles or loops in the network.

Deep Feedforward Networks (MLP = Multi-Layer Perceptrons)

Instead of manually building up deep network architectures, many python libraries are available online to help users to easily implement deep learning models.

- Trax (built on TensorFlow)
- Tensor2Tensor (built on TensorFlow)
- Pytorch
- Fast.ai ( built on top of PyTorch)
- Keras (built to be a higher level framework to interface with other frameworks like TensorFlow)
- Vowpal Wabbit 
- Theano
- Caffee 

Deep Learning Python libraries (frameworks)

What can convolutional neural networks be used for?

A generative adversarial network (GAN) is a class of machine learning framework where two neural networks, a generative model and a discriminant model, contest against each other in a "game" (in the form of a zero-sum game, where one agent's gain is another agent's loss). Given a training set, this technique learns to generate new data with the same statistics as the training set. 

The core idea of a GAN is based on the "indirect" training through the discriminator, which itself is also being updated dynamically. This basically means that the generator is not trained to minimize the distance to a specific image, but rather to fool the discriminator and the discriminator is rewarded every time it correctly deduces if a sample is from the model distribution or the data distribution. This enables the model to learn in an unsupervised manner.

Generative adversarial network(GAN)

Deep Learning models require incredible computational resources to successfully train a model. As a result, optimizing the training process is a key step in developing an algorithm. One approach involves finding the parameters of a neural network that minimize its loss function. This is accomplished via gradient descent, a popular technique for minimizing loss function error. 

Optimization for Training Deep Models

Whereas deep learning algorithms were once always implemented on a single computer's central processing unit (CPU), research has shown that families of CPUs or even a single graphics processing unit (GPU) is much more effective. Some systems employ even greater parallelization to achieve the fastest training times with distributed architectures.

Implementations of Deep Learning

Monte Carlo algorithms return results with variability. The variability can typically be reduced by allocating more resources. The benefit of Monte Carlo algorithms is that they can provide an approximated estimation with limited source while maintaining a certain degree of accuracy. In Machine Learning, many problems are too complicated to have precise answers. Monte Carlo approximations are therefore a good approach to these problems.

Monte Carlo Methods

Deep learning frameworks provide specialized software tools that streamline the construction, training, and deployment of neural networks. The availability of these efficient frameworks has made the design and implementation of whole system optimization significantly easier, which is a critical component in achieving high performance for complex deep learning pipelines.

Deep Learning Frameworks

An adversarial example is an input to a machine learning model that has been intentionally modified with small, often imperceptible perturbations designed to cause the model to output an incorrect classification. For instance, adding specific, calculated noise to an image of a panda can cause a deep neural network to misclassify it as a gibbon, even though the image remains visually identical to the original for a human observer.

Adversarial Example

It learns compositional representations for sentences recursively.

Recursive Neural Networks

It can better handle long sequential data.

Recurrent Neural Networks (RNNs)

GNNs are a general framework for defining deep neural networks on graph data. The goal of GNNs is to generate representations of nodes that actually depend on the structure of the graph, as well as any feature information available.

Graph Neural Networks (GNNs)

It utilizes attention mechanism to aggregate global relational information.

Attention-based Neural Networks

Different from SRE models, NRE mainly utilizes word embeddings and position embeddings instead of hand-craft features as inputs. Word embeddings are the most used input representations in NLP, which encode the semantic meaning of
words into vectors.

Word Embeddings

In order to capture the entity information in text, position embeddings are introduced to specify the relative distances between words and entities.

Position Embeddings

GNNs use knowledge graphs to generate the entity and relation embeddings.

Encoding Models using Graph Neural Networks (GNNs)

Various neural networks feed entities or relations or both to deep neural networks and computes a semantic matching score.

Encoding Models using Neural Networks

Recurrent Neural Networks (RNNs) can be used as encoding models to capture long-term dependencies within knowledge graphs, generating semantic embeddings that represent the interactions between entities and relations.

Encoding Models using Recurrent Neural Networks

Transformers use contextual information from the knowledge graph to generate the embeddings for entities and relations.

Encoding Models using Transformers

Factorization methods formulated KRL models as three-way tensor $\mathcal{X}$ decomposition.

Encoding Models using Factorisation Models

Linear/Bilinear Models encode the interaction of entities and relations using linear/bilinear operations.

Learn Before

Related

Learn After