Learn Before
Theory

Eigenspace Decomposition of Convex Quadratic Optimization

When optimizing a convex quadratic objective function, the positive definite matrix Q\mathbf{Q} can be decomposed into its eigensystem as Q=OopΛO\mathbf{Q} = \mathbf{O}^ op \boldsymbol{\Lambda} \mathbf{O}, where O\mathbf{O} is an orthogonal matrix and Λ\boldsymbol{\Lambda} is a diagonal matrix of strictly positive eigenvalues. By applying the change of variables z=O(xQ1c)\mathbf{z} = \mathbf{O} (\mathbf{x} - \mathbf{Q}^{-1} \mathbf{c}), the optimization problem is recast into a much simpler coordinate system that aligns with the eigenvectors of Q\mathbf{Q}. A key theorem demonstrates that when expressed in this eigensystem, both standard gradient descent and gradient descent with momentum do not mix different eigenspaces. Instead, the multi-dimensional optimization problem decomposes entirely into independent, coordinate-wise optimization processes along the directions of the eigenvectors.

0

1

Updated 2026-05-15

Contributors are:

Who are from:

Tags

D2L

Dive into Deep Learning @ D2L