Learn Before
Performance Advantage of Built-in Batch Normalization Layers
When constructing batch-normalized networks, using the built-in batch normalization layers provided by deep learning frameworks (e.g., nn.LazyBatchNorm2d and nn.LazyBatchNorm1d in PyTorch, nn.BatchNorm in MXNet and Flax, or tf.keras.layers.BatchNormalization in TensorFlow) yields a significant runtime advantage over custom Python implementations. The high-level API variant executes considerably faster because the framework's internal code has been compiled to optimized C++ or CUDA, whereas a hand-written implementation must be interpreted line by line by the Python runtime. Despite producing functionally equivalent behavior and accepting the same hyperparameters, the compiled built-in layers also simplify the user-facing code by automatically inferring the correct input dimensions, eliminating the need for the programmer to manually specify them.
0
1
Tags
D2L
Dive into Deep Learning @ D2L