In deep learning frameworks, the default summation function reduces a tensor along all of its axes, ultimately producing a single scalar value that represents the total sum of its elements. For example, in PyTorch, MXNet, and JAX, this is executed using the `.sum()` method on the tensor object (e.g., `x.sum()`), whereas TensorFlow provides the explicit `tf.reduce_sum()` function.

Claude

Often, it is necessary to calculate the total sum of a tensor's elements mathematically. For a vector $$\mathbf{x}$$ of length $$n$$, the sum of its elements is expressed as $$\sum_{i=1}^n x_i$$. To express sums over tensors of arbitrary shape, the summation simply extends over all of its axes. For example, the sum of all elements in an $$m \times n$$ matrix $$\mathbf{A}$$ is written using multiple summations as $$\sum_{i=1}^{m} \sum_{j=1}^{n} a_{ij}$$. Programmatically, summing all the elements within a tensor yields a new tensor containing only a single element.

Tensor Element Summation

Dive into Deep Learning

A related mathematical quantity to the sum is the mean, which is also called the average. The mean of a tensor is calculated by taking the sum of its elements and dividing it by the total number of elements contained in the tensor.

Tensor Mean

Given two vectors $$\mathbf{x}, \mathbf{y} \in \mathbb{R}^d$$, their dot product, commonly denoted as $$\mathbf{x}^	op \mathbf{y}$$ and also referred to as the inner product $$\langle \mathbf{x}, \mathbf{y} angle$$, is computed as the sum of the products of their elements at corresponding positions. This fundamental operation results in a scalar value and is mathematically defined by the formula: $$\mathbf{x}^	op \mathbf{y} = \sum_{i=1}^{d} x_i y_i$$.

Dot Product

Programmatic Tensor Summation

Instead of reducing an entire tensor to a single scalar, mathematical reduction operations like summation can be targeted along specific axes. For instance, summing an $$m \times n$$ matrix along its rows (axis $$0$$) eliminates the row dimension, yielding an output vector. Similarly, summing along the column dimension (axis $$1$$) produces a vector by adding elements across columns. Reducing a tensor along all of its axes simultaneously is mathematically equivalent to computing the total sum of all its elements.

Tensor Reduction along an Axis

To calculate the running total of a tensor's elements along a specific axis, one can perform a cumulative summation. Unlike standard reduction operations that collapse elements into a single value and remove the target axis, a cumulative sum computes a sequence of partial sums along the dimension. By design, this mathematical operation does not reduce the input tensor along any axis, producing an output tensor that shares the exact same shape as the input.

Learn Before

Related