A data scientist is creating a simple computer model to represent different fruits. The model must represent each fruit using three numerical features, always in this specific order: (1) average weight in grams, (2) sugar content in grams, and (3) water content in grams. How would this model represent an apple that has an average weight of 150g, a sugar content of 19g, and a water content of 129g?

A computer program needs to store the coordinates of a point in a 3-dimensional space. The coordinates are an x-value, a y-value, and a z-value, and their order is crucial (x, then y, then z). Which of the following mathematical structures is the most appropriate and direct way to represent these coordinates?

Analyze the two data structures described in the case study below. Which one can be directly represented as a mathematical vector? Justify your answer by explaining the key properties that a data structure must have to be considered a vector.

Identifying Vector Representations in Data

A vector, denoted by a bold lowercase letter such as $$\mathbf{x}$$, can be defined by explicitly listing its elements. This component-wise representation is an ordered sequence of scalars, commonly written as $$\mathbf{x} = (x_1, x_2, \dots, x_n)$$, where $$n$$ represents the vector's dimensionality.

Component-wise Notation for a Vector

A vector can be conceptualized as a fixed-length array of scalars. Mathematically and computationally, vectors are implemented as $$1^{	extrm{st}}$$-order tensors. When vectors represent data from real-world datasets, their scalar values hold real-world significance, such as a vector representing a patient where its components correspond to specific vital signs.

Google

Michigan State University

Claude

The linear algebra necessary to further explain Machine Learning in our capacity, requires a basic understanding of 
- Linear Systems of Equations
- Linear Transformations
- Vector Spaces
- Orthogonality
- Eigenvalues


Linear Algebra

A tensor is a generic mathematical framework for describing multidimensional arrays of numerical values, extending the concept of vectors and matrices to an arbitrary number of axes. The specific terminology depends on the number of axes: a one-dimensional tensor is a vector, a two-dimensional tensor is a matrix, and a tensor with $$n > 2$$ axes is referred to as an $$n^{	extrm{th}}$$-order tensor. Each individual scalar value within the tensor is called an element.

Tensor

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Reference of Foundations of Large Language Models Course

Dive into Deep Learning

Bretscher, O. (2019). Linear algebra with applications. Upper Saddle River, NJ: Pearson Education, Inc.

[Link to GitHub repo](https://github.com/gomson/Linear-Algebra/blob/master/Linear%20Algebra%20with%20Applications%205th%20Edition%20Otto%20Bretsch%20.pdf) 

Linear Algebra with Applications

Matrix is a two dimensional arrangement of numbers.

Linear Algebra - Matrices

To get the transpose of a matrix, we should do a mirror symmetry operation on the matrix according to one of its specific diagonal line, called the main diagonal.  The gotten matrix should be denoted as $A^T$.

Transpose

Matrices can be multiplied, with each value in the resulting matrix defined by the function
$C_{i,j} = \sum_k A_{i,k} B_{k,j}$

The size of the matrix is found by the height of the first node and the width of the second node. 

Matrix Multiplication

Used similarly to an inverse, but for nonsquare matrices. Can find rough values for systems with no real solution. Often denoted as $A^\dagger$ or "A dagger".

Given $A$ is an $m x n$ matrix and $A \vec{x} = \vec{y}$, the pseudoinverse of $A$ is the matrix $B$ such that $\vec{x} = B \vec{y}$.

Formal Definition: $A^\dagger = \lim_{\alpha \to 0} (A^T A + \alpha I )^{-1} A^T$
Practical Formula: $A^\dagger = V D^\dagger U^T$, where $U, D,$ and $V$ are the singular value decomposition of A, and the pseudoinverse, $D^\dagger$ of a diagonal matrix, $D$, is found by taking the reciprocal of its nonzero elements, then taking the transpose of the resulting elements.

Moore-Penrose Pseudoinverse

timeparticle. (2009, April 8). Using the Moore-Penrose Pseudoinverse to Solve Linear Equations [Video]. YouTube. https://youtu.be/5bxsxM2UTb4

Using the Moore-Penrose Pseudoinverse to Solve Linear Equations

The trace of a matrix is the sum of all values along its main diagonal, denoted $tr(A)$, where $A$ is an $m x m$ matrix. This may only be used for square matrices.

Traces have various properties:
$tr(A) = tr(A^T)$
$tr(ABC) = tr(CAB) = tr(BCA)$
For $A \in \mathbb{R}^{m x n} and B \in \mathbb{R}^{n x m}$, $tr(AB) = tr(BA)$. 

Example:
$tr(\begin{bmatrix} 2 & 8 & 3\\ 1 & 4 & 6\\ 7 & 9 & 5 \end{bmatrix}) = 2 + 4 + 5 = 11$

Linear Algebra (Trace)

The determinant of a matrix, $A$, is the product of $A$'s eigenvalues, where $A$ is an $m x m$ matrix. It has numerous applications in finding volume and using integrals. It can only be used on square matrices.

Determinants of different sizes:
$det(\begin{bmatrix} a \end{bmatrix}) = a$
$det(\begin{bmatrix} a & b\\ c & d \end{bmatrix}) = ad-bc$
$det(\begin{bmatrix} a & b & c\\ d & e & f\\ g & h & i \end{bmatrix}) = a(ei-fh) - b(di-fg) + c(dh-eg)$

Linear Algebra (Determinant)

A matrix satisfying that have nonzero entries only along the main diagonal is called diagonal matrix. Formally, a matrix $D$ is a diagonal matrix if and only if $D_{i,j} = 0$ for all $i \neq j$.
$diag(v)$ is used to denote a diagonal matrix which main diagonal is the vector $v$.
Keep in mind that not all diagonal matrices need to be square.

Linear Algebra - Diagonal Matrices

A vector with unit norm is called a unit vector.

Linear Algebra - Unit Vector

Vector $x$ and $y$ are orthogonal means that $x^Ty=0$

Linear Algebra - orthogonal

If the vectors are not only orthogonal but also unit vectors, we call them orthonormal.

Linear Algebra - orthonormal

A matrix is orthonormal matrix if and only if its rows are mutually orthonormal and its columns are mutually orthonormal as well.

Linear Algebra - orthogonal matrix

A vector $v$ is an eigenvector of a square matrix $A$ if and only if it's a nonzero vector which  multiplication by $A$ alters only the scale of $v$:
$Av = \lambda v$

Linear Algebra - eigenvector

Eigenvalue is the value $\lambda$ corresponding to the eigenvector which satisfies 
$Av = \lambda v$

Linear Algebra - eigenvalue

Suppose that a matrix A has n linearly independent eigenvectors {$v^{(1)}$, . . . , $v^{(n)}$} with corresponding eigenvalues {$\lambda _1$, . . . , $\lambda _n$}. We may concatenate all the eigenvectors to form a matrix V with one eigenvector per column: V = [$v^{(1)}$ , . . . , $v^{(n)}$]. Similarly, we can get a vector containing eigenvalues, naming it as $\lambda$. The eigendecomposition of A is then given by
$A = Vdiag(\lambda)V^{-1}$

Linear Algebra - eigendecomposition

Singular value decomposition (SVD) is an unsupervised dimensionality reduction technique used for feature extraction.  The singular value decomposition of an m x n matrix M is a factorization of the form U$$\sum$$M*.  U is an m x m unitary matrix. $$\sum$$ is a m x n diagonal matrix. V is a v x v unitary matrix.
Image link: [www.wikiwand.com/en/Singular_value_decomposition](https://www.wikiwand.com/en/Singular_value_decomposition)
 

Singular value decomposition (SVD)

The dot product can be conducted between two matrices if they have the same dimension. The dot product C=AB can be computed as $C_{i,j} =A_{i,:}B_{:,j}$.
Multiplication rules:
A(B+C)=AB+AC
A(BC)=(AB)C

Linear Algebra - Dot Product and Multiplication Rules

An n-dimensional identity matrix $I_{n}$ is a matrix that has ones on the diagonal and zeros as the rest of the elements.
The inverse of a matrix A can be defined as $A^{-1}$ such that $A^{-1}$A=$I_{n}$
We can use matrix inverse to solve a $Ax=B$: $x=BA^{-1}$

Linear Algebra - Identity and Inverse Matrices

A linear combination of a set of vectors {$V_{1}$,$V_{2}$,,...,$V_{n}$} is defined as $\sum_{i}$ $c_{i}V_{i}$ where $c_{i}$ are coefficents.
The span of a set of vectors is all the vectors that could be obtained through its linear combination.
A set of vectors is linearly independent if no vector is a linear combination of the other vectors in the set.

Linear dependence and span

In machine learning, we use norm to measure the size of a vector. An $L^{p}$ norm is defined as $||x||_{p}$=($\sum_{i}$ $|x_{i}|^{p}$)$^{1/p}$
Common norms such as L1 and L2 norm can be obtained by substituting p=1 and p=2 into the above formula. There are also some other norms such as max norm:
$||x||$=max$_{i}$ $|x_{i}|$

Linear Algebra - Norm

A standard basis vector, often denoted as $\vec{e}_k$, is a vector within a given vector space that has a value of 1 in its k-th coordinate and 0 in all other coordinates. These vectors form a basis, meaning any vector in the space can be represented as a linear combination of them.

Standard Basis Vector

The notation `a = (a, ..., a, a)` is used to represent a tuple or vector where every element is identical. This format signifies an ordered sequence composed entirely of the same element, in this case 'a', enclosed in parentheses.

Notation for a Tuple of Identical Elements

The memory state, denoted as Mem, is represented as a tuple. The first element of the tuple is the arithmetic mean of key vectors ($\mathbf{k}_j$) and the second element is the arithmetic mean of value vectors ($\mathbf{v}_j$), both summed from index $j=0$ to $i$. The formula is: $$Mem = \left( \frac{\sum_{j=0}^{i} \mathbf{k}_j}{i + 1}, \frac{\sum_{j=0}^{i} \mathbf{v}_j}{i + 1} \right)$$ This calculation effectively summarizes the information contained in the sequence of key-value pairs up to step $i$.

Memory State as an Average of Keys and Values

A sequence of variables is commonly denoted as $x = (x_1, x_2, \dots, x_n)$ or simply $x_1, x_2, \dots, x_n$. This represents an ordered collection of elements, where each variable $x_i$ is identified by its index $i$. Unlike a set, the order of elements in a sequence is significant, making it a foundational structure for representing data like time series, text, or the components of a vector.

Notation for a Sequence of Variables

A matrix is a rectangular array of numbers logically arranged in horizontal rows and vertical columns. In the context of multidimensional data arrays, a matrix is formally defined as a $$2^{\textrm{nd}}$$-order tensor, meaning it is a tensor with exactly two axes. It provides a structured mathematical format to compactly organize data and is frequently used to represent systems of linear equations.

Matrix

An element-wise product (also known as the Hadamard product) of two matrices with the same shape, such as $$A_{m,n}$$ and $$B_{m,n}$$, results in a new matrix of identical shape. Each element in the resulting matrix is the product of the elements at the corresponding positions in matrices $$A$$ and $$B$$. This is a specific instance of a binary elementwise tensor operation.

Element-wise Product

When performing elementwise binary operations on two tensors that do not share the same shape, deep learning frameworks utilize a broadcasting mechanism to make their dimensions compatible. This procedure involves two sequential steps: first, it expands one or both tensors by replicating their elements along any axis with a length of $$1$$ so that both tensors acquire identical shapes. Second, the framework executes the desired elementwise operation on these newly expanded, uniformly shaped tensors.

Broadcasting Mechanism

Vector

Most everyday mathematics consists of manipulating numbers one at a time. Formally, these single numerical values are called scalars. For example, in the expression for converting temperature, $$c = \frac{5}{9}(f - 32)$$, the values $$5$$, $$9$$, and $$32$$ are constant scalars, whereas the variables $$c$$ and $$f$$ represent unknown scalars.

Scalars

A symmetric matrix is a specific subset of square matrices that are perfectly equal to their own transpose. Mathematically, a square matrix $$\mathbf{A}$$ is considered symmetric if and only if the condition $$\mathbf{A} = \mathbf{A}^	op$$ holds true. This implies that the matrix elements are mirror images across the main diagonal, meaning that $$a_{ij} = a_{ji}$$ for all valid row indices $$i$$ and column indices $$j$$.

Symmetric Matrix

In deep learning data representation, when data requires only a single axis, it is structured as a one-dimensional tensor. This specific form of a tensor is formally referred to as a vector.

Vector (1D Tensor)

Like elements in standard Python lists, the individual elements within a tensor can be accessed using indexing. Tensor indexing is zero-based, meaning the first element along any given axis is located at index $$0$$. It is important to contrast this programmatic convention with formal linear algebra, where vectors typically utilize one-based indexing (starting at subscript $$1$$).

Tensor Indexing

Converting a deep learning tensor to a NumPy array (`ndarray`), or vice versa, is straightforward but involves framework-specific memory management behaviors. In PyTorch, the converted objects share the same underlying memory (e.g., via `X.numpy()` and `torch.from_numpy()`), meaning an in-place modification to one affects the other. Conversely, in frameworks like TensorFlow (via `X.numpy()` and `tf.constant()`), MXNet (via `X.asnumpy()` and `np.array()`), and JAX (via `jax.device_get()` and `jax.device_put()`), the converted result does not share memory. This isolation prevents the CPU or GPU from halting computations while waiting for Python's NumPy package to interact with the shared memory chunk.

Tensor to NumPy Array Conversion

To convert a deep learning tensor containing a single element (a size-$$1$$ tensor) into a standard Python scalar, practitioners can invoke the `.item()` method or apply Python's built-in conversion functions, such as `float()` or `int()`. This operation extracts the underlying numerical value from the multidimensional array structure, making it available for native Python computations.

Size-1 Tensor to Python Scalar Conversion

For the JAX framework, the `jnp.array()` function (from the `jax.numpy` module) is used to cast standard arrays into JAX's specific array format. Notably, when converting standard floating-point NumPy arrays, JAX often defaults to creating a `float32` array rather than `float64` depending on the system configuration.

jax.numpy.array()

In mathematical notation, general tensors are typically denoted by capital letters with a special sans-serif font face, such as $$\mathsf{X}$$, $$\mathsf{Y}$$, and $$\mathsf{Z}$$. Their indexing mechanism naturally extends the subscript notation used for matrices; for example, an individual element within a $$3^{\textrm{rd}}$$-order tensor might be referenced as $$x_{ijk}$$ or $$[\mathsf{X}]_{1, 2i-1, 3}$$.

Typographical Conventions for General Tensors

In deep learning, a single color image is mathematically and programmatically represented as a $$3^{\textrm{rd}}$$-order tensor. The three distinct axes of this tensor correspond to the image's height, width, and color channels. At each spatial location defined by the height and width dimensions, the color intensities (such as red, green, and blue) are stacked along the channel axis.

Single Image Representation as a 3rd-Order Tensor

In deep learning frameworks, higher-order tensors (those with an order greater than two) are constructed programmatically by expanding the number of components specified in their shape tuple. Similar to the creation of matrices, a generic sequential array of elements can be rearranged into a multidimensional structure by passing the desired axis lengths to a function like reshape, such as .reshape(2, 3, 4) for a $$3^{	extrm{rd}}$$-order tensor.

Programmatic Construction of Higher-Order Tensors

Adding or multiplying a scalar and a tensor produces a new tensor with the exact same shape as the original tensor. In these operations, the scalar value is distributed elementwise, meaning it is independently added to or multiplied by every individual element within the tensor.

Tensor-Scalar Arithmetic

Multiple tensors can be combined into a single larger tensor by concatenating, or stacking them end-to-end. To perform this operation, one must provide a list of the tensors and specify the target axis along which they will be concatenated. In the resulting tensor, the length along the specified concatenation axis is equal to the sum of the lengths of all input tensors along that same axis, while the lengths along all other axes remain unchanged.

Tensor Concatenation

An elementwise tensor operation applies a standard scalar function to the individual elements of a tensor. For a unary scalar operator, denoted mathematically as $$f: \mathbb{R} \rightarrow \mathbb{R}$$, the function maps each real number in the tensor to another real number independently. For a binary scalar operator, denoted as $$f: \mathbb{R}, \mathbb{R} \rightarrow \mathbb{R}$$, the operation maps pairs of real numbers to a single real number. When a binary operator is applied to two tensors of identical shape—whether they are vectors, matrices, or higher-order tensors—the function evaluates each pair of corresponding elements to produce a new tensor that has the exact same shape as its operands.

Elementwise Tensor Operation

Often, it is necessary to calculate the total sum of a tensor's elements mathematically. For a vector $$\mathbf{x}$$ of length $$n$$, the sum of its elements is expressed as $$\sum_{i=1}^n x_i$$. To express sums over tensors of arbitrary shape, the summation simply extends over all of its axes. For example, the sum of all elements in an $$m \times n$$ matrix $$\mathbf{A}$$ is written using multiple summations as $$\sum_{i=1}^{m} \sum_{j=1}^{n} a_{ij}$$. Programmatically, summing all the elements within a tensor yields a new tensor containing only a single element.

Tensor Element Summation

The tensor class serves as the fundamental interface for storing and manipulating data within deep learning libraries. Tensors encapsulate a wide array of functionalities, providing support for construction routines, indexing and slicing, basic mathematical operations, broadcasting mechanisms, memory-efficient in-place assignments, and the ability to convert to and from other standard Python objects.

Tensor Class Interface Summary

In deep learning programming frameworks, it is frequently necessary to initialize tensors such that they contain exclusively zeros. This is programmatically accomplished using dedicated functions (typically named zeros) that accept a desired shape tuple and construct a new multidimensional array where every element is precisely set to $$0$$.

Tensor Initialization with Zeros

In addition to zero-initialization, deep learning frameworks provide functions (typically named ones) to construct tensors where every element is initialized to $$1$$. By supplying a shape tuple, a practitioner can instantiate a multidimensional array prepopulated entirely with $$1$$s, which is a common requirement for various baseline computations and matrix operations.

Tensor Initialization with Ones

A standard method for generating a prepopulated one-dimensional tensor (a vector) is to create a sequence of evenly spaced values. By invoking a function such as arange(n), the framework constructs a vector starting at $$0$$ (inclusive) and ending at $$n$$ (exclusive), with a default interval size of $$1$$. This is a primary technique for programmatic iteration and indexing.

Evenly Spaced Tensor Initialization

For many deep learning procedures, such as initializing the parameters of neural networks, tensor elements must be sampled randomly and independently from a specific probability distribution. A common programmatic operation is to generate a tensor of a given shape where the values are drawn from a standard Gaussian (normal) distribution, which has a mean of $$0$$ and a standard deviation of $$1$$.

Random Tensor Initialization

Tensors can be constructed explicitly by supplying the exact numerical literals for every element using nested Python lists. The hierarchical depth of the nested lists corresponds to the axes of the resulting tensor. For example, to construct a matrix, a list of lists is provided where the outermost list defines axis $$0$$ (the rows) and each inner list defines axis $$1$$ (the columns).

Programmatic Construction of Tensors from Nested Lists

In the context of deep learning programming, the term 'tensor' is commonly used to refer to software objects instantiated from the tensor class. These data structures are called tensors because they provide a programmatic mechanism to manage arrays with an arbitrary number of axes, analogous to mathematical tensors. While it can sometimes be confusing that the same term denotes both the formal mathematical entity and its software realization, the intended meaning is generally evident from the context.

Tensor as a Software Object

The principles of matrix decomposition can be generalized to apply to high-order tensors. In machine learning, these tensor decompositions are utilized to discover underlying, low-dimensional structures in complex datasets that possess more than two axes, enabling the solution of advanced prediction problems.

Tensor Decomposition

In deep learning, an element refers to any individual entry or scalar component within a given tensor. This terminology is used universally to describe the constituent parts of multidimensional arrays, whether they are representing the data within a layer or the parameters of a convolution kernel.

Learn Before

Related

Learn After