In PyTorch, a single GPU can be indicated by torch.device('cuda'). Unlike the CPU device, a GPU device only represents one specific graphics card and its corresponding memory, not all GPUs on the system.

GPU Device Allocation in PyTorch

The number of available GPUs on a system can be queried in PyTorch using the function torch.cuda.device_count().

Querying Available GPUs in PyTorch

PyTorch's CUDA support lets programs select a CUDA-capable GPU device, create or move tensors onto that device, and run supported tensor operations there using GPU acceleration. Tensors can also be moved between CPU and GPU devices.

University of Michigan - Ann Arbor

Google

An open source deep learning library that specializes in tensor computations and GPU acceleration. It allows you to build neural networks and has plenty of features such as implementing custom layers and network architectures.

PyTorch

One of the major features of PyTorch is how it represents data. It uses a multi-dimensional array like in NumPy but they are called Tensors. To a programmer the the initialization and manipulation of the array is the exact same or very similar to NumPy. But under the hood, they are used to dynamically generate a graph and apply automatic differentiation.

Tensors in PyTorch

CUDA in PyTorch

PyTorch package for implementing a variety of optimizations alogithms. Interface is abstracted well enough that any optimization method can be used without changing surrounding code. There's also the option to implement a custom optimization method. Some examples of ones already implemented by PyTorch are:
- Adadelta
- Adagrad
- Adam
- ASGD
- RMSprop
- SGD

torch.optim

When working with large datasets (which you usually do with neural networks), it takes a large amount of time to load it all into memory before training. Also since we generally use a GPU for neural networks, there can also be memory limitations of the system. PyTorch's dataloader parallelizes the data loading process with automatic batching which in turn boosts the speed and saves memory.

DataLoader

One of the most popular applications of deep learning is computer vision and torchvision is a library that goes hand in hand with PyTorch. It has efficient utilities for image and video transformations, model architectures, and some of the most popular datasets already loaded.

Torchvision

PyTorch supports several categories of neural-network layers, including convolution, pooling, padding, normalization, recurrent, transformer, linear, dropout, sparse, vision, shuffle, and DataParallel (multi-GPU) layers.

Layer Types in PyTorch

PyTorch Autograd performs automatic differentiation on computations represented by a dynamically generated graph. After .backward() is called on a loss tensor, Autograd traverses the graph backward, computes gradients for participating model parameters, and stores them in their .grad attributes. An optimizer can then use these gradients to update the parameters.

Learn Before

Related

Learn After