Inspecting a minibatch of data provides intuition about its structure. Each minibatch of features has a shape determined by the batch size and the dimensionality of the input features. Similarly, the corresponding minibatch of labels will have a matching shape given by the batch size. For instance, if the batch size is $$32$$ and features are two-dimensional, the features shape will be $$(32, 2)$$ and the labels shape will be $$(32, 1)$$.

Claude

A data batch, commonly referred to as a minibatch, is a discrete, smaller subset of a larger dataset that is processed simultaneously during a single iteration of machine learning model training or evaluation. Training typically requires multiple passes over a dataset, grabbing one minibatch at a time. Each minibatch consists of a tuple of features and corresponding labels. Processing data in minibatches rather than the entire dataset balances computational efficiency with hardware memory constraints.

Data Batch

Dive into Deep Learning

Inspecting Data Minibatches

Naive data iteration—which loads an entire dataset into memory and performs random memory access—is inefficient and can cause memory issues for real-world problems. Built-in iterators implemented in deep learning frameworks are considerably more efficient. They are designed to seamlessly handle large data sources, such as data stored in files, received via streams, or generated on the fly, without requiring all data to be loaded into memory simultaneously. Furthermore, these well-implemented data iterators exploit high-performance computing, such as utilizing GPUs for rapid image decompression, video transcoding, and other preprocessing tasks, to prevent data input/output from slowing down the overall training loop.

Efficiency of Deep Learning Framework Iterators

In an object-oriented deep learning architecture, the `prepare_batch` method is registered within a `Trainer` class to handle incoming minibatches before they are processed by the model. Although its default implementation may simply return the batch unchanged, it serves as an essential hook. This method can be customized by subclasses to perform preprocessing tasks or to dynamically move data to appropriate hardware accelerators before the batch is passed into the model's `training_step` or `validation_step`.

Learn Before

Related