Given two vectors $$\mathbf{x}, \mathbf{y} \in \mathbb{R}^d$$, their dot product, commonly denoted as $$\mathbf{x}^	op \mathbf{y}$$ and also referred to as the inner product $$\langle \mathbf{x}, \mathbf{y} angle$$, is computed as the sum of the products of their elements at corresponding positions. This fundamental operation results in a scalar value and is mathematically defined by the formula: $$\mathbf{x}^	op \mathbf{y} = \sum_{i=1}^{d} x_i y_i$$.

Dot Product

A norm is a mathematical function $$\| \cdot \|$$ that maps a vector to a scalar, informally measuring its overall size or magnitude. To be a valid norm, the function must satisfy three properties for any vectors $$\mathbf{x}$$ and $$\mathbf{y}$$ and scalar $$\alpha \in \mathbb{R}$$:

1. Scaled norm: $$\left\|\alpha \mathbf{x}ight\| = |\alpha| \left\|\mathbf{x}ight\|$$
2. Triangle inequality: $$\left\|\mathbf{x} + \mathbf{y}ight\| \leq \left\|\mathbf{x}ight\| + \left\|\mathbf{y}ight\|$$
3. Nonnegativity: $$\left\|\mathbf{x}ight\| > 0$$ for all $$\mathbf{x} 
eq 0$$.

Vector Norm

Vectorization is a computational technique used to replace explicit, element-by-element loops (such as Python for-loops) with optimized operations on entire data structures simultaneously. In machine learning, vectorization leverages highly optimized linear algebra libraries to perform mathematical operations—like calculating the elementwise sum of vectors or processing whole minibatches of examples—at once. This approach dramatically reduces execution time by yielding order-of-magnitude speedups, simplifies code, reduces the potential for bugs, and increases portability by offloading mathematics to underlying libraries.

Vectorization

In deep learning data representation, when data requires only a single axis, it is structured as a one-dimensional tensor. This specific form of a tensor is formally referred to as a vector.

Claude

A tensor is a generic mathematical framework for describing multidimensional arrays of numerical values, extending the concept of vectors and matrices to an arbitrary number of axes. The specific terminology depends on the number of axes: a one-dimensional tensor is a vector, a two-dimensional tensor is a matrix, and a tensor with $$n > 2$$ axes is referred to as an $$n^{	extrm{th}}$$-order tensor. Each individual scalar value within the tensor is called an element.

Tensor

Dive into Deep Learning

Vector (1D Tensor)

Like elements in standard Python lists, the individual elements within a tensor can be accessed using indexing. Tensor indexing is zero-based, meaning the first element along any given axis is located at index $$0$$. It is important to contrast this programmatic convention with formal linear algebra, where vectors typically utilize one-based indexing (starting at subscript $$1$$).

Tensor Indexing

Converting a deep learning tensor to a NumPy array (`ndarray`), or vice versa, is straightforward but involves framework-specific memory management behaviors. In PyTorch, the converted objects share the same underlying memory (e.g., via `X.numpy()` and `torch.from_numpy()`), meaning an in-place modification to one affects the other. Conversely, in frameworks like TensorFlow (via `X.numpy()` and `tf.constant()`), MXNet (via `X.asnumpy()` and `np.array()`), and JAX (via `jax.device_get()` and `jax.device_put()`), the converted result does not share memory. This isolation prevents the CPU or GPU from halting computations while waiting for Python's NumPy package to interact with the shared memory chunk.

Tensor to NumPy Array Conversion

To convert a deep learning tensor containing a single element (a size-$$1$$ tensor) into a standard Python scalar, practitioners can invoke the `.item()` method or apply Python's built-in conversion functions, such as `float()` or `int()`. This operation extracts the underlying numerical value from the multidimensional array structure, making it available for native Python computations.

Size-1 Tensor to Python Scalar Conversion

For the JAX framework, the `jnp.array()` function (from the `jax.numpy` module) is used to cast standard arrays into JAX's specific array format. Notably, when converting standard floating-point NumPy arrays, JAX often defaults to creating a `float32` array rather than `float64` depending on the system configuration.

jax.numpy.array()

In mathematical notation, general tensors are typically denoted by capital letters with a special sans-serif font face, such as $$\mathsf{X}$$, $$\mathsf{Y}$$, and $$\mathsf{Z}$$. Their indexing mechanism naturally extends the subscript notation used for matrices; for example, an individual element within a $$3^{\textrm{rd}}$$-order tensor might be referenced as $$x_{ijk}$$ or $$[\mathsf{X}]_{1, 2i-1, 3}$$.

Typographical Conventions for General Tensors

In deep learning, a single color image is mathematically and programmatically represented as a $$3^{\textrm{rd}}$$-order tensor. The three distinct axes of this tensor correspond to the image's height, width, and color channels. At each spatial location defined by the height and width dimensions, the color intensities (such as red, green, and blue) are stacked along the channel axis.

Single Image Representation as a 3rd-Order Tensor

In deep learning frameworks, higher-order tensors (those with an order greater than two) are constructed programmatically by expanding the number of components specified in their shape tuple. Similar to the creation of matrices, a generic sequential array of elements can be rearranged into a multidimensional structure by passing the desired axis lengths to a function like reshape, such as .reshape(2, 3, 4) for a $$3^{	extrm{rd}}$$-order tensor.

Programmatic Construction of Higher-Order Tensors

Adding or multiplying a scalar and a tensor produces a new tensor with the exact same shape as the original tensor. In these operations, the scalar value is distributed elementwise, meaning it is independently added to or multiplied by every individual element within the tensor.

Tensor-Scalar Arithmetic

Multiple tensors can be combined into a single larger tensor by concatenating, or stacking them end-to-end. To perform this operation, one must provide a list of the tensors and specify the target axis along which they will be concatenated. In the resulting tensor, the length along the specified concatenation axis is equal to the sum of the lengths of all input tensors along that same axis, while the lengths along all other axes remain unchanged.

Tensor Concatenation

An elementwise tensor operation applies a standard scalar function to the individual elements of a tensor. For a unary scalar operator, denoted mathematically as $$f: \mathbb{R} \rightarrow \mathbb{R}$$, the function maps each real number in the tensor to another real number independently. For a binary scalar operator, denoted as $$f: \mathbb{R}, \mathbb{R} \rightarrow \mathbb{R}$$, the operation maps pairs of real numbers to a single real number. When a binary operator is applied to two tensors of identical shape—whether they are vectors, matrices, or higher-order tensors—the function evaluates each pair of corresponding elements to produce a new tensor that has the exact same shape as its operands.

Elementwise Tensor Operation

Often, it is necessary to calculate the total sum of a tensor's elements mathematically. For a vector $$\mathbf{x}$$ of length $$n$$, the sum of its elements is expressed as $$\sum_{i=1}^n x_i$$. To express sums over tensors of arbitrary shape, the summation simply extends over all of its axes. For example, the sum of all elements in an $$m \times n$$ matrix $$\mathbf{A}$$ is written using multiple summations as $$\sum_{i=1}^{m} \sum_{j=1}^{n} a_{ij}$$. Programmatically, summing all the elements within a tensor yields a new tensor containing only a single element.

Tensor Element Summation

The tensor class serves as the fundamental interface for storing and manipulating data within deep learning libraries. Tensors encapsulate a wide array of functionalities, providing support for construction routines, indexing and slicing, basic mathematical operations, broadcasting mechanisms, memory-efficient in-place assignments, and the ability to convert to and from other standard Python objects.

Tensor Class Interface Summary

A vector can be conceptualized as a fixed-length array of scalars. Mathematically and computationally, vectors are implemented as $$1^{	extrm{st}}$$-order tensors. When vectors represent data from real-world datasets, their scalar values hold real-world significance, such as a vector representing a patient where its components correspond to specific vital signs.

Vector

In deep learning programming frameworks, it is frequently necessary to initialize tensors such that they contain exclusively zeros. This is programmatically accomplished using dedicated functions (typically named zeros) that accept a desired shape tuple and construct a new multidimensional array where every element is precisely set to $$0$$.

Tensor Initialization with Zeros

In addition to zero-initialization, deep learning frameworks provide functions (typically named ones) to construct tensors where every element is initialized to $$1$$. By supplying a shape tuple, a practitioner can instantiate a multidimensional array prepopulated entirely with $$1$$s, which is a common requirement for various baseline computations and matrix operations.

Tensor Initialization with Ones

A standard method for generating a prepopulated one-dimensional tensor (a vector) is to create a sequence of evenly spaced values. By invoking a function such as arange(n), the framework constructs a vector starting at $$0$$ (inclusive) and ending at $$n$$ (exclusive), with a default interval size of $$1$$. This is a primary technique for programmatic iteration and indexing.

Evenly Spaced Tensor Initialization

For many deep learning procedures, such as initializing the parameters of neural networks, tensor elements must be sampled randomly and independently from a specific probability distribution. A common programmatic operation is to generate a tensor of a given shape where the values are drawn from a standard Gaussian (normal) distribution, which has a mean of $$0$$ and a standard deviation of $$1$$.

Random Tensor Initialization

Tensors can be constructed explicitly by supplying the exact numerical literals for every element using nested Python lists. The hierarchical depth of the nested lists corresponds to the axes of the resulting tensor. For example, to construct a matrix, a list of lists is provided where the outermost list defines axis $$0$$ (the rows) and each inner list defines axis $$1$$ (the columns).

Programmatic Construction of Tensors from Nested Lists

In the context of deep learning programming, the term 'tensor' is commonly used to refer to software objects instantiated from the tensor class. These data structures are called tensors because they provide a programmatic mechanism to manage arrays with an arbitrary number of axes, analogous to mathematical tensors. While it can sometimes be confusing that the same term denotes both the formal mathematical entity and its software realization, the intended meaning is generally evident from the context.

Tensor as a Software Object

The principles of matrix decomposition can be generalized to apply to high-order tensors. In machine learning, these tensor decompositions are utilized to discover underlying, low-dimensional structures in complex datasets that possess more than two axes, enabling the solution of advanced prediction problems.

Learn Before

Related

Learn After