A matrix is singular if and only if any of the eigenvalues are zero.

University of Michigan - Ann Arbor

Suppose that a matrix A has n linearly independent eigenvectors {$v^{(1)}$, . . . , $v^{(n)}$} with corresponding eigenvalues {$\lambda _1$, . . . , $\lambda _n$}. We may concatenate all the eigenvectors to form a matrix V with one eigenvector per column: V = [$v^{(1)}$ , . . . , $v^{(n)}$]. Similarly, we can get a vector containing eigenvalues, naming it as $\lambda$. The eigendecomposition of A is then given by
$A = Vdiag(\lambda)V^{-1}$

Linear Algebra - eigendecomposition

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Singular Matrix Judgement According to Eigenvalues

A matrix whose eigenvalues are all positive is called positive definite.

Positive Definite

A matrix whose eigenvalues are all positive or zero valued is called positive semidefi-
nite.

Positive Semidefinite

If all eigenvalues are negative, the matrix is negative definite.

Negative Definite

If all eigenvalues are negative or zero valued, it is negative semidefinite.

Negative Semidefinite

When optimizing a convex quadratic objective function, the positive definite matrix $$\mathbf{Q}$$ can be decomposed into its eigensystem as $$\mathbf{Q} = \mathbf{O}^	op \boldsymbol{\Lambda} \mathbf{O}$$, where $$\mathbf{O}$$ is an orthogonal matrix and $$\boldsymbol{\Lambda}$$ is a diagonal matrix of strictly positive eigenvalues. By applying the change of variables $$\mathbf{z} = \mathbf{O} (\mathbf{x} - \mathbf{Q}^{-1} \mathbf{c})$$, the optimization problem is recast into a much simpler coordinate system that aligns with the eigenvectors of $$\mathbf{Q}$$. A key theorem demonstrates that when expressed in this eigensystem, both standard gradient descent and gradient descent with momentum do not mix different eigenspaces. Instead, the multi-dimensional optimization problem decomposes entirely into independent, coordinate-wise optimization processes along the directions of the eigenvectors.

Learn Before

Related