CUDA helps keep track of which GPU is being used and ensures the tensors allocated will be on that device. It allows you to leverage GPU computational speed for certain tensor operations but also has functionality that allows you to move the tensor between the CPU and GPU.

University of Michigan - Ann Arbor

An open source deep learning library that specializes in tensor computations and GPU acceleration. It allows you to build neural networks and has plenty of features such as implementing custom layers and network architectures.

PyTorch

One of the major features of PyTorch is how it represents data. It uses a multi-dimensional array like in NumPy but they are called Tensors. To a programmer the the initialization and manipulation of the array is the exact same or very similar to NumPy. But under the hood, they are used to dynamically generate a graph and apply automatic differentiation.

Tensors in PyTorch

There are 4 main steps when training a neural network: assemble graph, forward propagation, compute loss, backward propagation. The most complex and computationally difficult one is the backward propagation. After calling .backward() on the loss tensor, Autograd automatically calculates and stores the gradients for each models parameter and stores it in a .grad attribute so it can be accessed. This makes training a model only take a few lines of code but still be customizable to a variety of loss functions and optimizers.

PyTorch Autograd

CUDA in PyTorch

The core building block of a neural network is a layer and PyTorch makes it so the type of hidden layer used is customizable. Types of layers available are:
- Convolution
- Pooling
- Padding
- Normalization
- Recurrent
- Transformer
- Linear
- Dropout
- Sparse
- Vision
- Shuffle
- DataParallel (multi-GPU)

Layer Types in PyTorch

PyTorch package for implementing a variety of optimizations alogithms. Interface is abstracted well enough that any optimization method can be used without changing surrounding code. There's also the option to implement a custom optimization method. Some examples of ones already implemented by PyTorch are:
- Adadelta
- Adagrad
- Adam
- ASGD
- RMSprop
- SGD

torch.optim

When working with large datasets (which you usually do with neural networks), it takes a large amount of time to load it all into memory before training. Also since we generally use a GPU for neural networks, there can also be memory limitations of the system. PyTorch's dataloader parallelizes the data loading process with automatic batching which in turn boosts the speed and saves memory.

DataLoader

One of the most popular applications of deep learning is computer vision and torchvision is a library that goes hand in hand with PyTorch. It has efficient utilities for image and video transformations, model architectures, and some of the most popular datasets already loaded.

Learn Before

Related