The principles of matrix decomposition can be generalized to apply to high-order tensors. In machine learning, these tensor decompositions are utilized to discover underlying, low-dimensional structures in complex datasets that possess more than two axes, enabling the solution of advanced prediction problems.

Tensor Decomposition

In machine learning, matrices can be decomposed into factors that reveal underlying, low-dimensional structures within real-world datasets. These matrix decompositions are a foundational tool used by various machine learning subfields to analyze data, discover latent patterns, and solve complex prediction problems.

Claude

A matrix is a rectangular array of numbers logically arranged in horizontal rows and vertical columns. In the context of multidimensional data arrays, a matrix is formally defined as a $$2^{\textrm{nd}}$$-order tensor, meaning it is a tensor with exactly two axes. It provides a structured mathematical format to compactly organize data and is frequently used to represent systems of linear equations.

Matrix

In the context of machine learning, matrices serve as the standard mathematical structure for representing tabular datasets. Typically, this data is logically organized such that each horizontal row of the matrix corresponds to an individual data record or sample, while each vertical column corresponds to a distinct feature or attribute associated with those records. This structured format enables entire datasets to be processed efficiently using parallelized linear algebra operations.

Dataset Representation using Matrices

Dive into Deep Learning

An augmented matrix provides a compact way to represent a system of linear equations. To create one, each equation must first be expressed in standard form. The variables' coefficients are organized into the left columns, while the constants are placed in the far-right column. A vertical line is inserted to replace the equal signs, visually separating the coefficients from the constants.

Augmented Matrix for a System of Equations

Every square matrix has a specific real number associated with it. This associated value is called the determinant of the square matrix.

Determinant of a Square Matrix

To find the determinant of a $$2 \times 2$$ square matrix $$\begin{bmatrix} a & b \\ c & d \end{bmatrix}$$, where $$a$$, $$b$$, $$c$$, and $$d$$ are real numbers, the matrix is first written with vertical bars instead of square brackets: $$\begin{vmatrix} a & b \\ c & d \end{vmatrix}$$. The real number value of this determinant is calculated by subtracting the product of the off-diagonal entries ($$bc$$) from the product of the main diagonal entries ($$ad$$):

$$\begin{vmatrix} a & b \\ c & d \end{vmatrix} = ad - bc$$

Determinant of a $$2 \times 2$$ Matrix

To evaluate the determinant of a $$2 \times 2$$ matrix, write the determinant with vertical lines, then subtract the products of its diagonals. For example, to evaluate the determinant of the matrix $$\begin{bmatrix} 4 & -2 \\ 3 & -1 \end{bmatrix}$$, first write it as $$\begin{vmatrix} 4 & -2 \\ 3 & -1 \end{vmatrix}$$. Next, subtract the product of the bottom-left and top-right entries from the product of the top-left and bottom-right entries:

$$(4)(-1) - (3)(-2)$$

Simplifying this expression yields $$-4 - (-6)$$, which equals $$-4 + 6 = 2$$. Thus, the determinant is $$2$$. Similarly, the determinant of $$\begin{bmatrix} -3 & -4 \\ -2 & 0 \end{bmatrix}$$ is evaluated as $$(-3)(0) - (-2)(-4)$$, which simplifies to $$0 - 8 = -8$$.

Example: Evaluating the Determinant of a $$2 \times 2$$ Matrix

Evaluate the determinant of the matrix $$\begin{bmatrix} 5 & -3 \\ 2 & -4 \end{bmatrix}$$.

**Solution:**

First, write the matrix in determinant form using vertical bars:

$$\begin{vmatrix} 5 & -3 \\ 2 & -4 \end{vmatrix}$$

Next, calculate the determinant by subtracting the product of the off-diagonal entries from the product of the main diagonal entries:

$$(5)(-4) - (2)(-3)$$

Now, simplify the expression by performing the multiplications:

$$-20 - (-6)$$

Finally, convert the subtraction to addition and solve:

$$-20 + 6 = -14$$

The determinant is $$-14$$.

Practice: Evaluating the Determinant of a $$2 \times 2$$ Matrix

The order of a matrix is defined by its dimensions, specifically its number of rows and columns. A matrix possessing $$m$$ rows and $$n$$ columns has an order of $$m 	imes n$$, which is often read as an '$$m$$ by $$n$$' matrix. For instance, a matrix containing $$2$$ rows and $$3$$ columns is designated as a $$2 	imes 3$$ matrix.

Order of a Matrix

A tensor is a generic mathematical framework for describing multidimensional arrays of numerical values, extending the concept of vectors and matrices to an arbitrary number of axes. The specific terminology depends on the number of axes: a one-dimensional tensor is a vector, a two-dimensional tensor is a matrix, and a tensor with $$n > 2$$ axes is referred to as an $$n^{	extrm{th}}$$-order tensor. Each individual scalar value within the tensor is called an element.

Tensor

When a multidimensional array requires more than two axes (where $$k > 2$$), the specialized names like vector or matrix are discarded. Instead, the data structure is formally referred to as a $$k^{	extrm{th}}$$-order tensor.

k-th Order Tensor (Axes/Dimensions)

In mathematical notation, matrices are conventionally denoted by bold capital letters, such as $$\mathbf{X}$$, $$\mathbf{Y}$$, and $$\mathbf{Z}$$. To specify the dimensions and the set from which the elements are drawn, the notation $$\mathbf{A} \in \mathbb{R}^{m 	imes n}$$ is used. This expression formally indicates that the matrix $$\mathbf{A}$$ is composed of $$m 	imes n$$ real-valued scalars, logically arranged into exactly $$m$$ rows and $$n$$ columns.

Typographical Conventions for Matrices

In deep learning frameworks, a mathematical matrix $$\mathbf{A} \in \mathbb{R}^{m \times n}$$ is programmatically implemented as a $$2^{\textrm{nd}}$$-order tensor possessing a shape of $$(m, n)$$. You can dynamically construct a matrix from an existing appropriately sized tensor by utilizing the `reshape` function (e.g., `.reshape(m, n)`). This operation reorganizes the sequential data into the specified row and column dimensions, allowing a generic one-dimensional tensor of $$m \times n$$ elements to be explicitly cast into a valid $$m \times n$$ matrix structure without altering the underlying numerical values.

Programmatic Representation of Matrices

Every individual number or scalar located within a matrix is mathematically referred to as an element or an entry of that matrix.

Element of a Matrix

A matrix $$\mathbf{A} \in \mathbb{R}^{m 	imes n}$$ can be visualized as a vertical stack of $$m$$ row vectors. Formally, it is expressed as:

$$\mathbf{A}= \begin{bmatrix} \mathbf{a}^	op_{1} \ \mathbf{a}^	op_{2} \ \vdots \ \mathbf{a}^	op_m \ \end{bmatrix}$$

where each $$\mathbf{a}^	op_{i} \in \mathbb{R}^n$$ is a row vector representing the $$i^	extrm{th}$$ row of the matrix $$\mathbf{A}$$.

Matrix Representation via Row Vectors

Often, it is necessary to calculate the total sum of a tensor's elements mathematically. For a vector $$\mathbf{x}$$ of length $$n$$, the sum of its elements is expressed as $$\sum_{i=1}^n x_i$$. To express sums over tensors of arbitrary shape, the summation simply extends over all of its axes. For example, the sum of all elements in an $$m \times n$$ matrix $$\mathbf{A}$$ is written using multiple summations as $$\sum_{i=1}^{m} \sum_{j=1}^{n} a_{ij}$$. Programmatically, summing all the elements within a tensor yields a new tensor containing only a single element.

Tensor Element Summation

A matrix $$\mathbf{B} \in \mathbb{R}^{k \times m}$$ can be visualized as a horizontal collection of $$m$$ column vectors. Formally, it is expressed as $$\mathbf{B}=\begin{bmatrix} \mathbf{b}_{1} & \mathbf{b}_{2} & \cdots & \mathbf{b}_{m} \end{bmatrix}$$, where each $$\mathbf{b}_{j} \in \mathbb{R}^k$$ is a column vector representing the $$j^\textrm{th}$$ column of the matrix $$\mathbf{B}$$.

Learn Before

Related

Learn After