The transformation $C(\mathbf{x}, t\theta)$ performs component-wise rotations on a vector in complex space. It is defined by the formula: $$C(\mathbf{x}, t\theta) = \sum_{k=1}^{d/2} x'_{k} e^{it\theta_k} \vec{e}_k$$ This equation represents the final rotated vector as a linear combination. Each term in the sum consists of a complex component $x'_{k}$ (derived from the original real vector $\mathbf{x}$) rotated by an angle $t\theta_k$, and then scaled by its corresponding standard basis vector $\vec{e}_k$. The term $\vec{e}_k$ represents a vector with a value of 1 in the k-th position and zeros elsewhere.

Component-wise Vector Rotation in Complex Space

A standard basis vector, often denoted as $\vec{e}_k$, is a vector within a given vector space that has a value of 1 in its k-th coordinate and 0 in all other coordinates. These vectors form a basis, meaning any vector in the space can be represented as a linear combination of them.

Google

The linear algebra necessary to further explain Machine Learning in our capacity, requires a basic understanding of 
- Linear Systems of Equations
- Linear Transformations
- Vector Spaces
- Orthogonality
- Eigenvalues


Linear Algebra

Reference of Foundations of Large Language Models Course

Bretscher, O. (2019). Linear algebra with applications. Upper Saddle River, NJ: Pearson Education, Inc.

[Link to GitHub repo](https://github.com/gomson/Linear-Algebra/blob/master/Linear%20Algebra%20with%20Applications%205th%20Edition%20Otto%20Bretsch%20.pdf) 

Linear Algebra with Applications

A singular number acting just as a value. They are written in lowercase.

Linear Algebra (Scalars)

Matrix is a two dimensional arrangement of numbers.

Linear Algebra - Matrices

Vectors are an array of numbers, such as scalars. They are written in lowercase with bolding.

Linear Algebra (Vectors)

To get the transpose of a matrix, we should do a mirror symmetry operation on the matrix according to one of its specific diagonal line, called the main diagonal.  The gotten matrix should be denoted as $A^T$.

Transpose

Broadcasting is a way to perform numerical calculations on tensors of different shapes. For example, $C = A+b$.
$A$ is a matrix and b is a vector. The vector $b$ is added to each row of the matrix. It's the simplest example of broadcasting.

Linear Algebra - broadcasting

Matrices can be multiplied, with each value in the resulting matrix defined by the function
$C_{i,j} = \sum_k A_{i,k} B_{k,j}$

The size of the matrix is found by the height of the first node and the width of the second node. 

Matrix Multiplication

An element-wise product of two matrices with shame shape, for example $A_{m,n}$ and $B_{m,n}$ should be a matrix with the same shape and it contains the para-position multiplication results of the matrices $A$ and $B$. It's called Hadamard product as well.

Element-wise Product

Used similarly to an inverse, but for nonsquare matrices. Can find rough values for systems with no real solution. Often denoted as $A^\dagger$ or "A dagger".

Given $A$ is an $m x n$ matrix and $A \vec{x} = \vec{y}$, the pseudoinverse of $A$ is the matrix $B$ such that $\vec{x} = B \vec{y}$.

Formal Definition: $A^\dagger = \lim_{\alpha \to 0} (A^T A + \alpha I )^{-1} A^T$
Practical Formula: $A^\dagger = V D^\dagger U^T$, where $U, D,$ and $V$ are the singular value decomposition of A, and the pseudoinverse, $D^\dagger$ of a diagonal matrix, $D$, is found by taking the reciprocal of its nonzero elements, then taking the transpose of the resulting elements.

Moore-Penrose Pseudoinverse

timeparticle. (2009, April 8). Using the Moore-Penrose Pseudoinverse to Solve Linear Equations [Video]. YouTube. https://youtu.be/5bxsxM2UTb4

Using the Moore-Penrose Pseudoinverse to Solve Linear Equations

The trace of a matrix is the sum of all values along its main diagonal, denoted $tr(A)$, where $A$ is an $m x m$ matrix. This may only be used for square matrices.

Traces have various properties:
$tr(A) = tr(A^T)$
$tr(ABC) = tr(CAB) = tr(BCA)$
For $A \in \mathbb{R}^{m x n} and B \in \mathbb{R}^{n x m}$, $tr(AB) = tr(BA)$. 

Example:
$tr(\begin{bmatrix} 2 & 8 & 3\\ 1 & 4 & 6\\ 7 & 9 & 5 \end{bmatrix}) = 2 + 4 + 5 = 11$

Linear Algebra (Trace)

The determinant of a matrix, $A$, is the product of $A$'s eigenvalues, where $A$ is an $m x m$ matrix. It has numerous applications in finding volume and using integrals. It can only be used on square matrices.

Determinants of different sizes:
$det(\begin{bmatrix} a \end{bmatrix}) = a$
$det(\begin{bmatrix} a & b\\ c & d \end{bmatrix}) = ad-bc$
$det(\begin{bmatrix} a & b & c\\ d & e & f\\ g & h & i \end{bmatrix}) = a(ei-fh) - b(di-fg) + c(dh-eg)$

Linear Algebra (Determinant)

A matrix satisfying that have nonzero entries only along the main diagonal is called diagonal matrix. Formally, a matrix $D$ is a diagonal matrix if and only if $D_{i,j} = 0$ for all $i \neq j$.
$diag(v)$ is used to denote a diagonal matrix which main diagonal is the vector $v$.
Keep in mind that not all diagonal matrices need to be square.

Linear Algebra - Diagonal Matrices

A matrix is a symmetric matrix if and only if it's equal to its own transpose.

Linear Algebra - Symmetric Matrix

A vector with unit norm is called a unit vector.

Linear Algebra - Unit Vector

Vector $x$ and $y$ are orthogonal means that $x^Ty=0$

Linear Algebra - orthogonal

If the vectors are not only orthogonal but also unit vectors, we call them orthonormal.

Linear Algebra - orthonormal

A matrix is orthonormal matrix if and only if its rows are mutually orthonormal and its columns are mutually orthonormal as well.

Linear Algebra - orthogonal matrix

A vector $v$ is an eigenvector of a square matrix $A$ if and only if it's a nonzero vector which  multiplication by $A$ alters only the scale of $v$:
$Av = \lambda v$

Linear Algebra - eigenvector

Eigenvalue is the value $\lambda$ corresponding to the eigenvector which satisfies 
$Av = \lambda v$

Linear Algebra - eigenvalue

Suppose that a matrix A has n linearly independent eigenvectors {$v^{(1)}$, . . . , $v^{(n)}$} with corresponding eigenvalues {$\lambda _1$, . . . , $\lambda _n$}. We may concatenate all the eigenvectors to form a matrix V with one eigenvector per column: V = [$v^{(1)}$ , . . . , $v^{(n)}$]. Similarly, we can get a vector containing eigenvalues, naming it as $\lambda$. The eigendecomposition of A is then given by
$A = Vdiag(\lambda)V^{-1}$

Linear Algebra - eigendecomposition

Singular value decomposition (SVD) is an unsupervised dimensionality reduction technique used for feature extraction.  The singular value decomposition of an m x n matrix M is a factorization of the form U$$\sum$$M*.  U is an m x m unitary matrix. $$\sum$$ is a m x n diagonal matrix. V is a v x v unitary matrix.
Image link: [www.wikiwand.com/en/Singular_value_decomposition](https://www.wikiwand.com/en/Singular_value_decomposition)
 

Singular value decomposition (SVD)

The dot product can be conducted between two matrices if they have the same dimension. The dot product C=AB can be computed as $C_{i,j} =A_{i,:}B_{:,j}$.
Multiplication rules:
A(B+C)=AB+AC
A(BC)=(AB)C

Linear Algebra - Dot Product and Multiplication Rules

An n-dimensional identity matrix $I_{n}$ is a matrix that has ones on the diagonal and zeros as the rest of the elements.
The inverse of a matrix A can be defined as $A^{-1}$ such that $A^{-1}$A=$I_{n}$
We can use matrix inverse to solve a $Ax=B$: $x=BA^{-1}$

Linear Algebra - Identity and Inverse Matrices

A linear combination of a set of vectors {$V_{1}$,$V_{2}$,,...,$V_{n}$} is defined as $\sum_{i}$ $c_{i}V_{i}$ where $c_{i}$ are coefficents.
The span of a set of vectors is all the vectors that could be obtained through its linear combination.
A set of vectors is linearly independent if no vector is a linear combination of the other vectors in the set.

Linear dependence and span

In machine learning, we use norm to measure the size of a vector. An $L^{p}$ norm is defined as $||x||_{p}$=($\sum_{i}$ $|x_{i}|^{p}$)$^{1/p}$
Common norms such as L1 and L2 norm can be obtained by substituting p=1 and p=2 into the above formula. There are also some other norms such as max norm:
$||x||$=max$_{i}$ $|x_{i}|$

Linear Algebra - Norm

Tensors are numbers having more than two axes.

Linear Algebra(Tensors)

A tuple or vector, representing an ordered collection of elements, is commonly denoted by enclosing its elements in parentheses. This notation can be used for sequences of identical elements, such as a vector of beta values, which would be written as $(\beta, \beta, \dots, \beta)$.

Tuple and Vector Notation

A vector, such as a $\frac{d}{2}$-dimensional vector denoted by $x$, can be represented by explicitly listing its individual scalar components. This component-wise representation is typically written as a sequence, for example, $x = (x_1, x_2, \dots, x_{d/2})$.

Learn Before

Related

Learn After