Learn Before
Eigenspace Decomposition of Convex Quadratic Optimization
When optimizing a convex quadratic objective function, the positive definite matrix can be decomposed into its eigensystem as , where is an orthogonal matrix and is a diagonal matrix of strictly positive eigenvalues. By applying the change of variables , the optimization problem is recast into a much simpler coordinate system that aligns with the eigenvectors of . A key theorem demonstrates that when expressed in this eigensystem, both standard gradient descent and gradient descent with momentum do not mix different eigenspaces. Instead, the multi-dimensional optimization problem decomposes entirely into independent, coordinate-wise optimization processes along the directions of the eigenvectors.
0
1
Tags
D2L
Dive into Deep Learning @ D2L