A related mathematical quantity to the sum is the mean, which is also called the average. The mean of a tensor is calculated by taking the sum of its elements and dividing it by the total number of elements contained in the tensor.

Tensor Mean

Given two vectors $$\mathbf{x}, \mathbf{y} \in \mathbb{R}^d$$, their dot product, commonly denoted as $$\mathbf{x}^	op \mathbf{y}$$ and also referred to as the inner product $$\langle \mathbf{x}, \mathbf{y} angle$$, is computed as the sum of the products of their elements at corresponding positions. This fundamental operation results in a scalar value and is mathematically defined by the formula: $$\mathbf{x}^	op \mathbf{y} = \sum_{i=1}^{d} x_i y_i$$.

Dot Product

In deep learning frameworks, the default summation function reduces a tensor along all of its axes, ultimately producing a single scalar value that represents the total sum of its elements. For example, in PyTorch, MXNet, and JAX, this is executed using the `.sum()` method on the tensor object (e.g., `x.sum()`), whereas TensorFlow provides the explicit `tf.reduce_sum()` function.

Programmatic Tensor Summation

Instead of reducing an entire tensor to a single scalar, mathematical reduction operations like summation can be targeted along specific axes. For instance, summing an $$m \times n$$ matrix along its rows (axis $$0$$) eliminates the row dimension, yielding an output vector. Similarly, summing along the column dimension (axis $$1$$) produces a vector by adding elements across columns. Reducing a tensor along all of its axes simultaneously is mathematically equivalent to computing the total sum of all its elements.

Tensor Reduction along an Axis

To calculate the running total of a tensor's elements along a specific axis, one can perform a cumulative summation. Unlike standard reduction operations that collapse elements into a single value and remove the target axis, a cumulative sum computes a sequence of partial sums along the dimension. By design, this mathematical operation does not reduce the input tensor along any axis, producing an output tensor that shares the exact same shape as the input.

Cumulative Tensor Summation

Often, it is necessary to calculate the total sum of a tensor's elements mathematically. For a vector $$\mathbf{x}$$ of length $$n$$, the sum of its elements is expressed as $$\sum_{i=1}^n x_i$$. To express sums over tensors of arbitrary shape, the summation simply extends over all of its axes. For example, the sum of all elements in an $$m \times n$$ matrix $$\mathbf{A}$$ is written using multiple summations as $$\sum_{i=1}^{m} \sum_{j=1}^{n} a_{ij}$$. Programmatically, summing all the elements within a tensor yields a new tensor containing only a single element.

Claude

A matrix is a rectangular array of numbers logically arranged in horizontal rows and vertical columns. In the context of multidimensional data arrays, a matrix is formally defined as a $$2^{\textrm{nd}}$$-order tensor, meaning it is a tensor with exactly two axes. It provides a structured mathematical format to compactly organize data and is frequently used to represent systems of linear equations.

Matrix

A tensor is a generic mathematical framework for describing multidimensional arrays of numerical values, extending the concept of vectors and matrices to an arbitrary number of axes. The specific terminology depends on the number of axes: a one-dimensional tensor is a vector, a two-dimensional tensor is a matrix, and a tensor with $$n > 2$$ axes is referred to as an $$n^{	extrm{th}}$$-order tensor. Each individual scalar value within the tensor is called an element.

Tensor

Dive into Deep Learning

An augmented matrix provides a compact way to represent a system of linear equations. To create one, each equation must first be expressed in standard form. The variables' coefficients are organized into the left columns, while the constants are placed in the far-right column. A vertical line is inserted to replace the equal signs, visually separating the coefficients from the constants.

Augmented Matrix for a System of Equations

Every square matrix has a specific real number associated with it. This associated value is called the determinant of the square matrix.

Determinant of a Square Matrix

To find the determinant of a $$2 \times 2$$ square matrix $$\begin{bmatrix} a & b \\ c & d \end{bmatrix}$$, where $$a$$, $$b$$, $$c$$, and $$d$$ are real numbers, the matrix is first written with vertical bars instead of square brackets: $$\begin{vmatrix} a & b \\ c & d \end{vmatrix}$$. The real number value of this determinant is calculated by subtracting the product of the off-diagonal entries ($$bc$$) from the product of the main diagonal entries ($$ad$$):

$$\begin{vmatrix} a & b \\ c & d \end{vmatrix} = ad - bc$$

Determinant of a $$2 \times 2$$ Matrix

To evaluate the determinant of a $$2 \times 2$$ matrix, write the determinant with vertical lines, then subtract the products of its diagonals. For example, to evaluate the determinant of the matrix $$\begin{bmatrix} 4 & -2 \\ 3 & -1 \end{bmatrix}$$, first write it as $$\begin{vmatrix} 4 & -2 \\ 3 & -1 \end{vmatrix}$$. Next, subtract the product of the bottom-left and top-right entries from the product of the top-left and bottom-right entries:

$$(4)(-1) - (3)(-2)$$

Simplifying this expression yields $$-4 - (-6)$$, which equals $$-4 + 6 = 2$$. Thus, the determinant is $$2$$. Similarly, the determinant of $$\begin{bmatrix} -3 & -4 \\ -2 & 0 \end{bmatrix}$$ is evaluated as $$(-3)(0) - (-2)(-4)$$, which simplifies to $$0 - 8 = -8$$.

Example: Evaluating the Determinant of a $$2 \times 2$$ Matrix

Evaluate the determinant of the matrix $$\begin{bmatrix} 5 & -3 \\ 2 & -4 \end{bmatrix}$$.

**Solution:**

First, write the matrix in determinant form using vertical bars:

$$\begin{vmatrix} 5 & -3 \\ 2 & -4 \end{vmatrix}$$

Next, calculate the determinant by subtracting the product of the off-diagonal entries from the product of the main diagonal entries:

$$(5)(-4) - (2)(-3)$$

Now, simplify the expression by performing the multiplications:

$$-20 - (-6)$$

Finally, convert the subtraction to addition and solve:

$$-20 + 6 = -14$$

The determinant is $$-14$$.

Practice: Evaluating the Determinant of a $$2 \times 2$$ Matrix

The order of a matrix is defined by its dimensions, specifically its number of rows and columns. A matrix possessing $$m$$ rows and $$n$$ columns has an order of $$m 	imes n$$, which is often read as an '$$m$$ by $$n$$' matrix. For instance, a matrix containing $$2$$ rows and $$3$$ columns is designated as a $$2 	imes 3$$ matrix.

Order of a Matrix

When a multidimensional array requires more than two axes (where $$k > 2$$), the specialized names like vector or matrix are discarded. Instead, the data structure is formally referred to as a $$k^{	extrm{th}}$$-order tensor.

k-th Order Tensor (Axes/Dimensions)

In mathematical notation, matrices are conventionally denoted by bold capital letters, such as $$\mathbf{X}$$, $$\mathbf{Y}$$, and $$\mathbf{Z}$$. To specify the dimensions and the set from which the elements are drawn, the notation $$\mathbf{A} \in \mathbb{R}^{m 	imes n}$$ is used. This expression formally indicates that the matrix $$\mathbf{A}$$ is composed of $$m 	imes n$$ real-valued scalars, logically arranged into exactly $$m$$ rows and $$n$$ columns.

Typographical Conventions for Matrices

In deep learning frameworks, a mathematical matrix $$\mathbf{A} \in \mathbb{R}^{m \times n}$$ is programmatically implemented as a $$2^{\textrm{nd}}$$-order tensor possessing a shape of $$(m, n)$$. You can dynamically construct a matrix from an existing appropriately sized tensor by utilizing the `reshape` function (e.g., `.reshape(m, n)`). This operation reorganizes the sequential data into the specified row and column dimensions, allowing a generic one-dimensional tensor of $$m \times n$$ elements to be explicitly cast into a valid $$m \times n$$ matrix structure without altering the underlying numerical values.

Programmatic Representation of Matrices

In the context of machine learning, matrices serve as the standard mathematical structure for representing tabular datasets. Typically, this data is logically organized such that each horizontal row of the matrix corresponds to an individual data record or sample, while each vertical column corresponds to a distinct feature or attribute associated with those records. This structured format enables entire datasets to be processed efficiently using parallelized linear algebra operations.

Dataset Representation using Matrices

Every individual number or scalar located within a matrix is mathematically referred to as an element or an entry of that matrix.

Element of a Matrix

A matrix $$\mathbf{A} \in \mathbb{R}^{m 	imes n}$$ can be visualized as a vertical stack of $$m$$ row vectors. Formally, it is expressed as:

$$\mathbf{A}= \begin{bmatrix} \mathbf{a}^	op_{1} \ \mathbf{a}^	op_{2} \ \vdots \ \mathbf{a}^	op_m \ \end{bmatrix}$$

where each $$\mathbf{a}^	op_{i} \in \mathbb{R}^n$$ is a row vector representing the $$i^	extrm{th}$$ row of the matrix $$\mathbf{A}$$.

Matrix Representation via Row Vectors

Tensor Element Summation

A matrix $$\mathbf{B} \in \mathbb{R}^{k \times m}$$ can be visualized as a horizontal collection of $$m$$ column vectors. Formally, it is expressed as $$\mathbf{B}=\begin{bmatrix} \mathbf{b}_{1} & \mathbf{b}_{2} & \cdots & \mathbf{b}_{m} \end{bmatrix}$$, where each $$\mathbf{b}_{j} \in \mathbb{R}^k$$ is a column vector representing the $$j^\textrm{th}$$ column of the matrix $$\mathbf{B}$$.

Matrix Representation via Column Vectors

In machine learning, matrices can be decomposed into factors that reveal underlying, low-dimensional structures within real-world datasets. These matrix decompositions are a foundational tool used by various machine learning subfields to analyze data, discover latent patterns, and solve complex prediction problems.

Matrix Decomposition

In deep learning data representation, when data requires only a single axis, it is structured as a one-dimensional tensor. This specific form of a tensor is formally referred to as a vector.

Vector (1D Tensor)

Like elements in standard Python lists, the individual elements within a tensor can be accessed using indexing. Tensor indexing is zero-based, meaning the first element along any given axis is located at index $$0$$. It is important to contrast this programmatic convention with formal linear algebra, where vectors typically utilize one-based indexing (starting at subscript $$1$$).

Tensor Indexing

Converting a deep learning tensor to a NumPy array (`ndarray`), or vice versa, is straightforward but involves framework-specific memory management behaviors. In PyTorch, the converted objects share the same underlying memory (e.g., via `X.numpy()` and `torch.from_numpy()`), meaning an in-place modification to one affects the other. Conversely, in frameworks like TensorFlow (via `X.numpy()` and `tf.constant()`), MXNet (via `X.asnumpy()` and `np.array()`), and JAX (via `jax.device_get()` and `jax.device_put()`), the converted result does not share memory. This isolation prevents the CPU or GPU from halting computations while waiting for Python's NumPy package to interact with the shared memory chunk.

Tensor to NumPy Array Conversion

To convert a deep learning tensor containing a single element (a size-$$1$$ tensor) into a standard Python scalar, practitioners can invoke the `.item()` method or apply Python's built-in conversion functions, such as `float()` or `int()`. This operation extracts the underlying numerical value from the multidimensional array structure, making it available for native Python computations.

Size-1 Tensor to Python Scalar Conversion

For the JAX framework, the `jnp.array()` function (from the `jax.numpy` module) is used to cast standard arrays into JAX's specific array format. Notably, when converting standard floating-point NumPy arrays, JAX often defaults to creating a `float32` array rather than `float64` depending on the system configuration.

jax.numpy.array()

In mathematical notation, general tensors are typically denoted by capital letters with a special sans-serif font face, such as $$\mathsf{X}$$, $$\mathsf{Y}$$, and $$\mathsf{Z}$$. Their indexing mechanism naturally extends the subscript notation used for matrices; for example, an individual element within a $$3^{\textrm{rd}}$$-order tensor might be referenced as $$x_{ijk}$$ or $$[\mathsf{X}]_{1, 2i-1, 3}$$.

Typographical Conventions for General Tensors

In deep learning, a single color image is mathematically and programmatically represented as a $$3^{\textrm{rd}}$$-order tensor. The three distinct axes of this tensor correspond to the image's height, width, and color channels. At each spatial location defined by the height and width dimensions, the color intensities (such as red, green, and blue) are stacked along the channel axis.

Single Image Representation as a 3rd-Order Tensor

In deep learning frameworks, higher-order tensors (those with an order greater than two) are constructed programmatically by expanding the number of components specified in their shape tuple. Similar to the creation of matrices, a generic sequential array of elements can be rearranged into a multidimensional structure by passing the desired axis lengths to a function like reshape, such as .reshape(2, 3, 4) for a $$3^{	extrm{rd}}$$-order tensor.

Programmatic Construction of Higher-Order Tensors

Adding or multiplying a scalar and a tensor produces a new tensor with the exact same shape as the original tensor. In these operations, the scalar value is distributed elementwise, meaning it is independently added to or multiplied by every individual element within the tensor.

Tensor-Scalar Arithmetic

Multiple tensors can be combined into a single larger tensor by concatenating, or stacking them end-to-end. To perform this operation, one must provide a list of the tensors and specify the target axis along which they will be concatenated. In the resulting tensor, the length along the specified concatenation axis is equal to the sum of the lengths of all input tensors along that same axis, while the lengths along all other axes remain unchanged.

Tensor Concatenation

An elementwise tensor operation applies a standard scalar function to the individual elements of a tensor. For a unary scalar operator, denoted mathematically as $$f: \mathbb{R} \rightarrow \mathbb{R}$$, the function maps each real number in the tensor to another real number independently. For a binary scalar operator, denoted as $$f: \mathbb{R}, \mathbb{R} \rightarrow \mathbb{R}$$, the operation maps pairs of real numbers to a single real number. When a binary operator is applied to two tensors of identical shape—whether they are vectors, matrices, or higher-order tensors—the function evaluates each pair of corresponding elements to produce a new tensor that has the exact same shape as its operands.

Elementwise Tensor Operation

The tensor class serves as the fundamental interface for storing and manipulating data within deep learning libraries. Tensors encapsulate a wide array of functionalities, providing support for construction routines, indexing and slicing, basic mathematical operations, broadcasting mechanisms, memory-efficient in-place assignments, and the ability to convert to and from other standard Python objects.

Tensor Class Interface Summary

A vector can be conceptualized as a fixed-length array of scalars. Mathematically and computationally, vectors are implemented as $$1^{	extrm{st}}$$-order tensors. When vectors represent data from real-world datasets, their scalar values hold real-world significance, such as a vector representing a patient where its components correspond to specific vital signs.

Vector

In deep learning programming frameworks, it is frequently necessary to initialize tensors such that they contain exclusively zeros. This is programmatically accomplished using dedicated functions (typically named zeros) that accept a desired shape tuple and construct a new multidimensional array where every element is precisely set to $$0$$.

Tensor Initialization with Zeros

In addition to zero-initialization, deep learning frameworks provide functions (typically named ones) to construct tensors where every element is initialized to $$1$$. By supplying a shape tuple, a practitioner can instantiate a multidimensional array prepopulated entirely with $$1$$s, which is a common requirement for various baseline computations and matrix operations.

Tensor Initialization with Ones

A standard method for generating a prepopulated one-dimensional tensor (a vector) is to create a sequence of evenly spaced values. By invoking a function such as arange(n), the framework constructs a vector starting at $$0$$ (inclusive) and ending at $$n$$ (exclusive), with a default interval size of $$1$$. This is a primary technique for programmatic iteration and indexing.

Evenly Spaced Tensor Initialization

For many deep learning procedures, such as initializing the parameters of neural networks, tensor elements must be sampled randomly and independently from a specific probability distribution. A common programmatic operation is to generate a tensor of a given shape where the values are drawn from a standard Gaussian (normal) distribution, which has a mean of $$0$$ and a standard deviation of $$1$$.

Random Tensor Initialization

Tensors can be constructed explicitly by supplying the exact numerical literals for every element using nested Python lists. The hierarchical depth of the nested lists corresponds to the axes of the resulting tensor. For example, to construct a matrix, a list of lists is provided where the outermost list defines axis $$0$$ (the rows) and each inner list defines axis $$1$$ (the columns).

Programmatic Construction of Tensors from Nested Lists

In the context of deep learning programming, the term 'tensor' is commonly used to refer to software objects instantiated from the tensor class. These data structures are called tensors because they provide a programmatic mechanism to manage arrays with an arbitrary number of axes, analogous to mathematical tensors. While it can sometimes be confusing that the same term denotes both the formal mathematical entity and its software realization, the intended meaning is generally evident from the context.

Tensor as a Software Object

The principles of matrix decomposition can be generalized to apply to high-order tensors. In machine learning, these tensor decompositions are utilized to discover underlying, low-dimensional structures in complex datasets that possess more than two axes, enabling the solution of advanced prediction problems.

Learn Before

Related

Learn After