In deep learning, distances and sizes are frequently used to formulate optimization objectives, and these are often mathematically expressed as norms. For example, norms are used to minimize the distance between predictions and ground truth observations, or to maximize the distance between representations of distinct entities (like photos of different people).

Norms as Objectives in Deep Learning

The $$\ell_2$$ norm, also known as the Euclidean norm, measures the Euclidean length of a vector. It is calculated as the square root of the sum of the squares of the vector's elements. For a vector $$\mathbf{x}$$ with $$n$$ elements, the $$\ell_2$$ norm is formally expressed as:
$$\left\|\mathbf{x}\right\|_2 = \sqrt{\sum_{i=1}^n x_i^2}$$

L2 Norm

Norms, which quantify the magnitude of a vector, are commonly applied to compute the distance between two vectors. This is achieved by calculating the norm of the difference between the two vectors, which mathematically measures how far apart they are in the vector space. For example, the distance between vectors $$\mathbf{u}$$ and $$\mathbf{v}$$ is computed as $$\lVert\mathbf{u} - \mathbf{v}\rVert$$.

Distance Between Vectors via Norms

A norm is a mathematical function $$\| \cdot \|$$ that maps a vector to a scalar, informally measuring its overall size or magnitude. To be a valid norm, the function must satisfy three properties for any vectors $$\mathbf{x}$$ and $$\mathbf{y}$$ and scalar $$\alpha \in \mathbb{R}$$:

1. Scaled norm: $$\left\|\alpha \mathbf{x}ight\| = |\alpha| \left\|\mathbf{x}ight\|$$
2. Triangle inequality: $$\left\|\mathbf{x} + \mathbf{y}ight\| \leq \left\|\mathbf{x}ight\| + \left\|\mathbf{y}ight\|$$
3. Nonnegativity: $$\left\|\mathbf{x}ight\| > 0$$ for all $$\mathbf{x} 
eq 0$$.

Claude

In deep learning data representation, when data requires only a single axis, it is structured as a one-dimensional tensor. This specific form of a tensor is formally referred to as a vector.

Vector (1D Tensor)

Dive into Deep Learning

Given two vectors $$\mathbf{x}, \mathbf{y} \in \mathbb{R}^d$$, their dot product, commonly denoted as $$\mathbf{x}^	op \mathbf{y}$$ and also referred to as the inner product $$\langle \mathbf{x}, \mathbf{y} angle$$, is computed as the sum of the products of their elements at corresponding positions. This fundamental operation results in a scalar value and is mathematically defined by the formula: $$\mathbf{x}^	op \mathbf{y} = \sum_{i=1}^{d} x_i y_i$$.

Dot Product

Vector Norm

Vectorization is a computational technique used to replace explicit, element-by-element loops (such as Python for-loops) with optimized operations on entire data structures simultaneously. In machine learning, vectorization leverages highly optimized linear algebra libraries to perform mathematical operations—like calculating the elementwise sum of vectors or processing whole minibatches of examples—at once. This approach dramatically reduces execution time by yielding order-of-magnitude speedups, simplifies code, reduces the potential for bugs, and increases portability by offloading mathematics to underlying libraries.

Learn Before

Related

Learn After