- Ease of programming (development and deployment)
- Running Speed
- Truly open (open source with good governance)
- Popularity

Criterion for Choosing a Deep Learning Framework

Deep Learning Frameworks References

Keras is an API designed for human beings, not machines. Keras follows best practices for reducing cognitive load: it offers consistent & simple APIs, it minimizes the number of user actions required for common use cases, and it provides clear & actionable error messages. It also has extensive documentation and developer guides.

https://keras.io/

Keras

TensorFlow is an end-to-end open source platform for machine learning. It has a comprehensive, flexible ecosystem of tools, libraries and community resources that lets researchers push the state-of-the-art in ML and developers easily build and deploy ML powered applications.

https://www.tensorflow.org/

TensorFlow

PyTorch is an open source machine learning framework that accelerates the path from research prototyping to production deployment.

https://pytorch.org/

PyTorch

Theano is a Python library that allows you to define, optimize, and efficiently evaluate mathematical expressions involving multi-dimensional arrays. It is built on top of NumPy.

https://pypi.org/project/Theano/

Theano

PaddlePaddle, as the only independent R&D deep learning platform in China, has been officially open-sourced to professional communities since 2016. It is an industrial platform with advanced technologies and rich features that cover core deep learning frameworks, basic model libraries, end-to-end development kits, tools & components as well as service platforms. PaddlePaddle is originated from industrial practices with dedication and commitments to industrialization. It has been widely adopted by a wide range of sectors including manufacturing, agriculture, enterprise service, and so on while serving more than 2.3 million developers. With such advantages, PaddlePaddle has helped an increasing number of partners commercialize AI.

https://pypi.org/project/paddlepaddle/

PaddlePaddle

A truly open source deep learning framework suited for flexible research prototyping and production.

https://mxnet.apache.org/


mxnet

Lasagne is a lightweight library to build and train neural networks in Theano.

https://lasagne.readthedocs.io/

Lasagne

Eclipse Deeplearning4j is the first commercial-grade, open-source, distributed deep-learning library written for Java and Scala. Integrated with Hadoop and Apache Spark, DL4J brings AI to business environments for use on distributed GPUs and CPUs.

https://deeplearning4j.org/

DL4J

The Microsoft Cognitive Toolkit (CNTK) is an open-source toolkit for commercial-grade distributed deep learning. It describes neural networks as a series of computational steps via a directed graph. CNTK allows the user to easily realize and combine popular model types such as feed-forward DNNs, convolutional neural networks (CNNs) and recurrent neural networks (RNNs/LSTMs). CNTK implements stochastic gradient descent (SGD, error backpropagation) learning with automatic differentiation and parallelization across multiple GPUs and servers.

https://docs.microsoft.com/en-us/cognitive-toolkit/

CNTK

Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by Berkeley AI Research (BAIR) and by community contributors. Yangqing Jia created the project during his PhD at UC Berkeley. Caffe is released under the BSD 2-Clause license.

https://caffe.berkeleyvision.org/

Caffe

1. Style

If you’re a Python programmer, then PyTorch will feel easy to pick up. It works the way you’d expect it to, right out of the box.

On the other hand, more coding languages are supported in TensorFlow than in PyTorch, which has a C++ API. You can use TensorFlow in both JavaScript and Swift. If you don’t want to write much low-level code, then Keras abstracts away a lot of the details for common use cases so you can build TensorFlow models without sweating the details.

2. Data and Model

What models are you using? If you want to use a specific pretrained model, like BERT or DeepDream, then you should research what it’s compatible with. Some pretrained models are available in only one library or the other, and some are available on both. The Model Garden and the PyTorch and TensorFlow hubs are also good resources to check.

What data do you need? If you want to use preprocessed data, then it may already be built into one library or the other. Check the docs to see—it will make your development go faster!

3. Project Goal

Where will your model live? If you want to deploy a model on mobile devices, then TensorFlow is a good bet because of TensorFlow Lite and its Swift API. For serving models, TensorFlow has tight integration with Google Cloud, but PyTorch is integrated into TorchServe on AWS. If you want to enter Kaggle competitions, then Keras will let you quickly iterate over experiments.

Think about these questions and examples at the outset of your project. Nail down the two or three most important components, and either TensorFlow or PyTorch will emerge as the right choice.

https://realpython.com/pytorch-vs-tensorflow/

PyTorch vs TensorFlow

High-level APIs in deep learning frameworks offer significant convenience by abstracting complex and potentially unstable operations, such as numerical stability, and allowing users to implement models concisely. This abstraction dramatically increases accessibility, enabling individuals without advanced statistical training to utilize deep learning. However, relying solely on high-level APIs presents drawbacks: it discourages the creation of custom, non-standard components and leaves practitioners unequipped to debug edge cases when the framework's abstractions fail. To invent novel architectures that frameworks do not natively support, deep learning practitioners must understand both the fundamental, from-scratch implementations and the concise, framework-based versions.

Trade-offs of High-Level APIs in Deep Learning

Deep learning frameworks provide specialized software tools that streamline the construction, training, and deployment of neural networks. The availability of these efficient frameworks has made the design and implementation of whole system optimization significantly easier, which is a critical component in achieving high performance for complex deep learning pipelines.

Claude

University of Michigan - Ann Arbor

Deep learning algorithms can be implemented to solve real-world problems today.

Deep feedforward networks, also called multilayer perceptrons (MLPs), is the quintessential deep learning model that is behind nearly all modern practical applications of deep learning.

Scaling these models to large inputs such as high-resolution images or long temporal sequences requires specialization. 
 - Convolutional neural networks (CNN) can be used for processing large images.
 - Recurrent neural networks (RNN) can be used for processing temporal sequences. 


Deep Learning Algorithms

A deep learning model is a powerful type of machine learning model differentiated by its architecture, which consists of many successive transformations of data chained together from top to bottom. This depth of continuous data transformations allows these models to tackle complex tasks by automatically learning multi-level representations. By doing so, deep learning effectively replaces not only the labor-intensive process of manual feature engineering, but also the shallow models that are typically used at the end of traditional machine learning pipelines.

Deep Learning Model

Dive into Deep Learning

A helpful website that introduces neural networks:
https://missinglink.ai/guides/neural-network-concepts/

Neural Network Reference

Convolutional networks, also known as convolutional neural networks, or CNNs, are a specialized kind of neural network for processing data that has a known grid-like topology. Examples include time-series data, which can be thought of as a 1-D grid taking samples at regular time intervals, and image data, which can be thought of as a 2-D grid of pixels. 

The network employs a mathematical operation called convolution. Convolution is a specialized kind of linear operation. Convolutional networks are simply neural networks that use convolution in place of general matrix multiplication in at least one of their layers.

The CNN gets its power from the following concepts:
- Sparse connectivity
- Parameter sharing

Convolutional Neural Network (CNN)

Recurrent neural networks, or RNNs, are a family of neural networks for processing sequential data. A recurrent neural network is specialized for processing a sequence of inputs $x^{(1)}, . . . , x^{(τ)}$ and each time add additional layers of comprehension on top of the previous inputs. 


Recurrent Neural Network (RNN)

**Generative Models**  As might be clear from their name , generative models are able to generate new contents by learning features of certain datasets. As for more formal definition, generative models specify in terms of a probabilistic model describing how it thinks the dataset was produced.  Using sampling from this probability distribution, we can create new data in the style of the old.
 
The joint probability distribution P(X,Y) is learned from the data, and then divided by P(X) to obtain the conditional probability distribution P(Y|X), which is used as a predictive model. It is called a generative model because the model represents the relationship between a given input X and an output Y. Common generative models are: Naive Bayes, Hidden Markov Model, Gaussian Mixture Model, Document Topic Generation Model (LDA), and restricted Boltzmann machine.

Generative Models

There are functions you can compute with a small layer that shallower networks would need a lot more hidden units in order to compute. 

Circuit Theory

https://www.simplilearn.com/deep-learning-algorithms-article

More about deep learning algorithms 

Data inputted into deep learning algorithms should be split into a distribution in which the majority of data (well over 70%+) is fed into the training set. This differs from traditional machine learning algorithms which had ~70% of the data given to the training set. This difference comes from the fact that there is now much more data available and deep learning algorithms improve with more data, whereas traditional machine learning algorithms improvement levels out after a certain amount of data is fed in.

For example, in a dataset of 1 million examples, you mighht decide that 10k examples are enough to evaluate which algorithm is better - 98% train, 1% dev, 1% test.

Deep learning train/dev/test split 

Deep feedforward networks, also called feedforward neural networks, or multilayer perceptrons (MLPs), are the quintessential deep learning models.

It was the first and simplest type of artificial neural network devised. In this network, the information moves in only one direction, forward, from the input nodes, through the hidden nodes (if any) and to the output nodes. There are no cycles or loops in the network.

Deep Feedforward Networks (MLP = Multi-Layer Perceptrons)

Instead of manually building up deep network architectures, many python libraries are available online to help users to easily implement deep learning models.

- Trax (built on TensorFlow)
- Tensor2Tensor (built on TensorFlow)
- Pytorch
- Fast.ai ( built on top of PyTorch)
- Keras (built to be a higher level framework to interface with other frameworks like TensorFlow)
- Vowpal Wabbit 
- Theano
- Caffee 

Deep Learning Python libraries (frameworks)

What can convolutional neural networks be used for?

A generative adversarial network (GAN) is a class of machine learning framework where two neural networks, a generative model and a discriminant model, contest against each other in a "game" (in the form of a zero-sum game, where one agent's gain is another agent's loss). Given a training set, this technique learns to generate new data with the same statistics as the training set. 

The core idea of a GAN is based on the "indirect" training through the discriminator, which itself is also being updated dynamically. This basically means that the generator is not trained to minimize the distance to a specific image, but rather to fool the discriminator and the discriminator is rewarded every time it correctly deduces if a sample is from the model distribution or the data distribution. This enables the model to learn in an unsupervised manner.

Generative adversarial network(GAN)

Deep Learning models require incredible computational resources to successfully train a model. As a result, optimizing the training process is a key step in developing an algorithm. One approach involves finding the parameters of a neural network that minimize its loss function. This is accomplished via gradient descent, a popular technique for minimizing loss function error. 

Optimization for Training Deep Models

Whereas deep learning algorithms were once always implemented on a single computer's central processing unit (CPU), research has shown that families of CPUs or even a single graphics processing unit (GPU) is much more effective. Some systems employ even greater parallelization to achieve the fastest training times with distributed architectures.

Implementations of Deep Learning

Monte Carlo algorithms return results with variability. The variability can typically be reduced by allocating more resources. The benefit of Monte Carlo algorithms is that they can provide an approximated estimation with limited source while maintaining a certain degree of accuracy. In Machine Learning, many problems are too complicated to have precise answers. Monte Carlo approximations are therefore a good approach to these problems.

Monte Carlo Methods

Deep Learning Frameworks

An adversarial example is an input to a machine learning model that has been intentionally modified with small, often imperceptible perturbations designed to cause the model to output an incorrect classification. For instance, adding specific, calculated noise to an image of a panda can cause a deep neural network to misclassify it as a gibbon, even though the image remains visually identical to the original for a human observer.

Adversarial Example

Moore's law refers to the historical trend of consistently cheaper and more powerful computation. In the context of machine learning, this exponential growth in compute budget allowed statistical models to spend more computer cycles optimizing parameters, shifting the ideal paradigm from linear models to computationally intensive deep neural networks.

Moore's Law in Deep Learning

Kryder's law describes the rapid advancement and cost reduction of data storage technologies. This availability of inexpensive data storage was a fundamental prerequisite for the deep learning revolution, as it allowed for the accumulation and retention of the massive datasets required to train deep neural networks.

Kryder's Law in Deep Learning

Graphics Processing Units (GPUs) were originally engineered to render graphics for computer gaming, but their highly parallel architecture proved exceptionally well-suited for the matrix operations pervasive in neural networks. Significant progress in GPU computation has triggered much of the recent advancement in deep learning by making previously infeasible, large-scale algorithms highly accessible and efficient.

Graphics Processing Unit (GPU) in Deep Learning

Multi-stage designs, such as memory networks and the neural programmer-interpreter, enable statistical models to execute iterative reasoning. These architectures allow the internal state of a deep neural network to be modified repeatedly across multiple steps, mirroring how a traditional computer processor modifies memory to carry out a chain of computation.

Multi-stage Reasoning in Deep Learning

A Diffusion Model is a deep generative architecture that gradually constructs data samples from random noise. It works by learning the denoising process to reverse a mathematical forward diffusion process (which gradually adds random noise to data). Diffusion models have become highly effective for tasks like photorealistic text-to-image generation.

Diffusion Model

The development of self-driving vehicles is a major indicator of progress in artificial intelligence. While full autonomy is complex because it requires a system to perceive, reason, and incorporate traffic rules, companies have achieved partial autonomy. Currently, deep learning is primarily utilized for the visual perception aspect of these problems, while the remaining reasoning and rule-incorporation logic is heavily tuned by engineers.

Autonomous Vehicles in Deep Learning

A major catalyst for the recent rapid progress in deep learning has been the immense abundance of available training data. This widespread proliferation of datasets is primarily driven by the deployment of inexpensive digital sensors and the massive scale of internet applications, which continuously generate the vast amounts of information necessary to effectively train complex, multi-layered deep learning models.

Learn Before

Related

Learn After