One of the major features of PyTorch is how it represents data. It uses a multi-dimensional array like in NumPy but they are called Tensors. To a programmer the the initialization and manipulation of the array is the exact same or very similar to NumPy. But under the hood, they are used to dynamically generate a graph and apply automatic differentiation.

Tensors in PyTorch

There are 4 main steps when training a neural network: assemble graph, forward propagation, compute loss, backward propagation. The most complex and computationally difficult one is the backward propagation. After calling .backward() on the loss tensor, Autograd automatically calculates and stores the gradients for each models parameter and stores it in a .grad attribute so it can be accessed. This makes training a model only take a few lines of code but still be customizable to a variety of loss functions and optimizers.

PyTorch Autograd

CUDA helps keep track of which GPU is being used and ensures the tensors allocated will be on that device. It allows you to leverage GPU computational speed for certain tensor operations but also has functionality that allows you to move the tensor between the CPU and GPU.

CUDA in PyTorch

The core building block of a neural network is a layer and PyTorch makes it so the type of hidden layer used is customizable. Types of layers available are:
- Convolution
- Pooling
- Padding
- Normalization
- Recurrent
- Transformer
- Linear
- Dropout
- Sparse
- Vision
- Shuffle
- DataParallel (multi-GPU)

Layer Types in PyTorch

PyTorch package for implementing a variety of optimizations alogithms. Interface is abstracted well enough that any optimization method can be used without changing surrounding code. There's also the option to implement a custom optimization method. Some examples of ones already implemented by PyTorch are:
- Adadelta
- Adagrad
- Adam
- ASGD
- RMSprop
- SGD

torch.optim

When working with large datasets (which you usually do with neural networks), it takes a large amount of time to load it all into memory before training. Also since we generally use a GPU for neural networks, there can also be memory limitations of the system. PyTorch's dataloader parallelizes the data loading process with automatic batching which in turn boosts the speed and saves memory.

DataLoader

One of the most popular applications of deep learning is computer vision and torchvision is a library that goes hand in hand with PyTorch. It has efficient utilities for image and video transformations, model architectures, and some of the most popular datasets already loaded.

Torchvision

An open source deep learning library that specializes in tensor computations and GPU acceleration. It allows you to build neural networks and has plenty of features such as implementing custom layers and network architectures.

University of Michigan - Ann Arbor

Instead of manually building up deep network architectures, many python libraries are available online to help users to easily implement deep learning models.

- Trax (built on TensorFlow)
- Tensor2Tensor (built on TensorFlow)
- Pytorch
- Fast.ai ( built on top of PyTorch)
- Keras (built to be a higher level framework to interface with other frameworks like TensorFlow)
- Vowpal Wabbit 
- Theano
- Caffee 

Deep Learning Python libraries (frameworks)

Natural Language Processing with Sequence Models is the third course in the Natural Language Prcessing Specialization published by deeplearning.ai.

Coursera: Natural Language Processing with Sequence Models

- Ease of programming (development and deployment)
- Running speed
- Truly open (open source with good governance)

Learn Before

Related

Learn After