Deep learning libraries allow developers to specify the exact axes along which a tensor should be reduced when invoking summation functions. The dimension corresponding to the specified reduction axis is subsequently removed from the shape of the resulting output tensor. In PyTorch, MXNet, and JAX, this is achieved by providing the `axis` parameter (e.g., `A.sum(axis=0)` or `A.sum(axis=[0, 1])`), while TensorFlow utilizes the same parameter within its `tf.reduce_sum(A, axis=0)` function.

Programmatic Tensor Reduction along an Axis

Sometimes it is useful to keep the number of axes unchanged when calculating the sum or mean of a tensor. A non-reduction sum calculates the total along a specified axis but retains that axis with a size of $$1$$ in the output shape, rather than completely eliminating the dimension. Maintaining the original number of axes is critical when the resulting tensor will be used in subsequent mathematical operations that rely on the broadcasting mechanism.

Non-Reduction Tensor Summation

Instead of reducing an entire tensor to a single scalar, mathematical reduction operations like summation can be targeted along specific axes. For instance, summing an $$m \times n$$ matrix along its rows (axis $$0$$) eliminates the row dimension, yielding an output vector. Similarly, summing along the column dimension (axis $$1$$) produces a vector by adding elements across columns. Reducing a tensor along all of its axes simultaneously is mathematically equivalent to computing the total sum of all its elements.

Claude

Often, it is necessary to calculate the total sum of a tensor's elements mathematically. For a vector $$\mathbf{x}$$ of length $$n$$, the sum of its elements is expressed as $$\sum_{i=1}^n x_i$$. To express sums over tensors of arbitrary shape, the summation simply extends over all of its axes. For example, the sum of all elements in an $$m \times n$$ matrix $$\mathbf{A}$$ is written using multiple summations as $$\sum_{i=1}^{m} \sum_{j=1}^{n} a_{ij}$$. Programmatically, summing all the elements within a tensor yields a new tensor containing only a single element.

Tensor Element Summation

Dive into Deep Learning

A related mathematical quantity to the sum is the mean, which is also called the average. The mean of a tensor is calculated by taking the sum of its elements and dividing it by the total number of elements contained in the tensor.

Tensor Mean

Given two vectors $$\mathbf{x}, \mathbf{y} \in \mathbb{R}^d$$, their dot product, commonly denoted as $$\mathbf{x}^	op \mathbf{y}$$ and also referred to as the inner product $$\langle \mathbf{x}, \mathbf{y} angle$$, is computed as the sum of the products of their elements at corresponding positions. This fundamental operation results in a scalar value and is mathematically defined by the formula: $$\mathbf{x}^	op \mathbf{y} = \sum_{i=1}^{d} x_i y_i$$.

Dot Product

In deep learning frameworks, the default summation function reduces a tensor along all of its axes, ultimately producing a single scalar value that represents the total sum of its elements. For example, in PyTorch, MXNet, and JAX, this is executed using the `.sum()` method on the tensor object (e.g., `x.sum()`), whereas TensorFlow provides the explicit `tf.reduce_sum()` function.

Programmatic Tensor Summation

Tensor Reduction along an Axis

To calculate the running total of a tensor's elements along a specific axis, one can perform a cumulative summation. Unlike standard reduction operations that collapse elements into a single value and remove the target axis, a cumulative sum computes a sequence of partial sums along the dimension. By design, this mathematical operation does not reduce the input tensor along any axis, producing an output tensor that shares the exact same shape as the input.

Learn Before

Related

Learn After