Pictorial Representation of Dot Product

The tensor product of two coordinate vectors is termed as Outer product. 

Outer Product

In a simplified model for analyzing text, two words are represented by the numerical vectors A = [4, -2, 1] and B = [3, 5, 6]. To compute a similarity score, an operation is performed where the corresponding elements of the two vectors are multiplied together, and the resulting products are then summed to produce a single number. What is the final similarity score?

Using the information provided in the case study, calculate the student's final weighted grade. Your answer should clearly show the two numerical lists (vectors) you used and the calculation that results in the final single-number grade.

Calculating a Weighted Final Grade

Consider two numerical lists, `X = [1, 5, 3]` and `Y = [2, 4]`. The operation that multiplies corresponding elements and sums the results can be successfully performed on these two lists to produce a single numerical value.

You are given two lists of numbers representing a shopping trip. The first list, `Quantities = [3, 1, 2]`, represents the number of each item bought. The second list, `Prices = [2, 5, 4]`, represents the price of each corresponding item. To find the total cost of the shopping trip, you must perform a specific sequence of calculations. Arrange the following steps in the correct order to calculate the total cost.

The dot product of two vectors can be conceptually and practically understood as a two-step process: first, performing an elementwise multiplication between the two vectors to yield a new vector of the same length, and second, calculating the sum of all elements within that resulting vector.

Dot Product via Elementwise Multiplication and Summation

A fundamental application of the dot product is the computation of a weighted sum. Given a vector of values $$\mathbf{x} \in \mathbb{R}^n$$ and a corresponding vector of weights $$\mathbf{w} \in \mathbb{R}^n$$, the weighted sum of the elements in $$\mathbf{x}$$ based on the weights in $$\mathbf{w}$$ can be concisely represented as the dot product $$\mathbf{x}^	op \mathbf{w}$$.

Weighted Sum via Dot Product

The dot product inherently relates to the geometry of vectors. Specifically, if two vectors are normalized to have unit length, their dot product directly computes the cosine of the angle between them.

Cosine of Angle via Dot Product

Given two vectors $$\mathbf{x}, \mathbf{y} \in \mathbb{R}^d$$, their dot product, commonly denoted as $$\mathbf{x}^	op \mathbf{y}$$ and also referred to as the inner product $$\langle \mathbf{x}, \mathbf{y} angle$$, is computed as the sum of the products of their elements at corresponding positions. This fundamental operation results in a scalar value and is mathematically defined by the formula: $$\mathbf{x}^	op \mathbf{y} = \sum_{i=1}^{d} x_i y_i$$.

Claude

University of Michigan - Ann Arbor

Google

Vectorization is a computational technique used to replace explicit, element-by-element loops (such as Python for-loops) with optimized operations on entire data structures simultaneously. In machine learning, vectorization leverages highly optimized linear algebra libraries to perform mathematical operations—like calculating the elementwise sum of vectors or processing whole minibatches of examples—at once. This approach dramatically reduces execution time by yielding order-of-magnitude speedups, simplifies code, reduces the potential for bugs, and increases portability by offloading mathematics to underlying libraries.

Vectorization

In deep learning data representation, when data requires only a single axis, it is structured as a one-dimensional tensor. This specific form of a tensor is formally referred to as a vector.

Vector (1D Tensor)

Often, it is necessary to calculate the total sum of a tensor's elements mathematically. For a vector $$\mathbf{x}$$ of length $$n$$, the sum of its elements is expressed as $$\sum_{i=1}^n x_i$$. To express sums over tensors of arbitrary shape, the summation simply extends over all of its axes. For example, the sum of all elements in an $$m \times n$$ matrix $$\mathbf{A}$$ is written using multiple summations as $$\sum_{i=1}^{m} \sum_{j=1}^{n} a_{ij}$$. Programmatically, summing all the elements within a tensor yields a new tensor containing only a single element.

Tensor Element Summation

Reference of Foundations of Large Language Models Course

Dive into Deep Learning

In machine learning, one of the optimization purpose is to reduce the error and get the best parameters for the machine learning model. By using a vectorized implementation in an optimization algorithm, we can make the process of computation much faster. The code also becomes simpler and easy to debug. A basic example of optimization algorithm is gradient descent.

Why is vectorization important in Machine Learning?

Dot Product

The computational advantage of vectorization can be demonstrated by comparing the time taken to add two high-dimensional vectors. For instance, when adding two $$10,000$$-dimensional vectors of all ones, using an explicit element-by-element for-loop in Python is significantly slower than using the vectorized elementwise addition operator (e.g., $$+$$). The vectorized operation leverages optimized linear algebra libraries to perform the entire addition simultaneously, yielding an order-of-magnitude speedup and dramatically faster execution times compared to the loop-based method.

Example: Computational Speedup of Vectorized Addition

When training machine learning models, processing whole minibatches of examples simultaneously is highly desirable for computational efficiency. Achieving this concurrent processing efficiently requires vectorizing the calculations to leverage fast linear algebra libraries, rather than relying on sequentially executed and costly Python for-loops.

Vectorization for Minibatch Processing

Beyond computational speedups, vectorizing code yields significant software engineering advantages. By pushing complex mathematical operations to highly optimized linear algebra libraries, developers reduce the number of calculations they must write manually from scratch. Delegating these mathematics to libraries decreases the potential for implementation errors and increases the overall portability of the code.

Software Engineering Benefits of Vectorization

A norm is a mathematical function $$\| \cdot \|$$ that maps a vector to a scalar, informally measuring its overall size or magnitude. To be a valid norm, the function must satisfy three properties for any vectors $$\mathbf{x}$$ and $$\mathbf{y}$$ and scalar $$\alpha \in \mathbb{R}$$:

1. Scaled norm: $$\left\|\alpha \mathbf{x}ight\| = |\alpha| \left\|\mathbf{x}ight\|$$
2. Triangle inequality: $$\left\|\mathbf{x} + \mathbf{y}ight\| \leq \left\|\mathbf{x}ight\| + \left\|\mathbf{y}ight\|$$
3. Nonnegativity: $$\left\|\mathbf{x}ight\| > 0$$ for all $$\mathbf{x} 
eq 0$$.

Vector Norm

A related mathematical quantity to the sum is the mean, which is also called the average. The mean of a tensor is calculated by taking the sum of its elements and dividing it by the total number of elements contained in the tensor.

Tensor Mean

In deep learning frameworks, the default summation function reduces a tensor along all of its axes, ultimately producing a single scalar value that represents the total sum of its elements. For example, in PyTorch, MXNet, and JAX, this is executed using the `.sum()` method on the tensor object (e.g., `x.sum()`), whereas TensorFlow provides the explicit `tf.reduce_sum()` function.

Programmatic Tensor Summation

Instead of reducing an entire tensor to a single scalar, mathematical reduction operations like summation can be targeted along specific axes. For instance, summing an $$m \times n$$ matrix along its rows (axis $$0$$) eliminates the row dimension, yielding an output vector. Similarly, summing along the column dimension (axis $$1$$) produces a vector by adding elements across columns. Reducing a tensor along all of its axes simultaneously is mathematically equivalent to computing the total sum of all its elements.

Tensor Reduction along an Axis

To calculate the running total of a tensor's elements along a specific axis, one can perform a cumulative summation. Unlike standard reduction operations that collapse elements into a single value and remove the target axis, a cumulative sum computes a sequence of partial sums along the dimension. By design, this mathematical operation does not reduce the input tensor along any axis, producing an output tensor that shares the exact same shape as the input.

Learn Before

Related

Learn After