When applying the broadcasting mechanism to add a $$3\times1$$ matrix `a` and a $$1\times2$$ matrix `b`, a deep learning framework cannot perform a direct elementwise addition because their shapes do not match. Instead, broadcasting resolves this by producing a larger $$3\times2$$ matrix: it replicates matrix `a` along its columns and matrix `b` along its rows. Once both matrices are expanded to the identical $$3\times2$$ shape, their elements are successfully added together.

Claude

When performing elementwise binary operations on two tensors that do not share the same shape, deep learning frameworks utilize a broadcasting mechanism to make their dimensions compatible. This procedure involves two sequential steps: first, it expands one or both tensors by replicating their elements along any axis with a length of $$1$$ so that both tensors acquire identical shapes. Second, the framework executes the desired elementwise operation on these newly expanded, uniformly shaped tensors.

Broadcasting Mechanism

Dive into Deep Learning

Tensor Broadcasting Example

Because a non-reduction sum retains its original axes after summing along a specific dimension, the resulting tensor can be directly used in elementwise operations with the original tensor via broadcasting. For instance, dividing a matrix by its non-reduction row sum seamlessly triggers the broadcasting mechanism to expand the sum tensor, thereby generating a new matrix where the elements of each row mathematically sum up to $$1$$.

Learn Before

Related