Preconditioning is an optimization technique used to help with scale adjustment. By applying a mathematical transformation that equalizes the scales or normalizes the curvature across different dimensions, preconditioning makes the landscape of the objective function more uniform. This allows optimization algorithms to navigate high-dimensional spaces more efficiently without requiring complex, dimension-specific learning rate adjustments.

Preconditioning

In high-dimensional optimization problems, adjusting the learning rate becomes significantly complicated. Because an objective function can exhibit drastically different scales and curvatures along its various dimensions, a single global learning rate might be excessively large for some directions—leading to divergence—while simultaneously being too small for others, resulting in sluggish progress.

Claude

When the objective function maps a $$d$$-dimensional vector $$\mathbf{x} = [x_1, x_2, \ldots, x_d]^	op$$ to a scalar, i.e., $$f: \mathbb{R}^d 	o \mathbb{R}$$, its gradient becomes a vector of $$d$$ partial derivatives:

$$
abla f(\mathbf{x}) = \left[\frac{\partial f(\mathbf{x})}{\partial x_1}, \frac{\partial f(\mathbf{x})}{\partial x_2}, \ldots, \frac{\partial f(\mathbf{x})}{\partial x_d}ight]^	op$$

Each component $$\partial f(\mathbf{x})/\partial x_i$$ captures the rate at which $$f$$ changes with respect to $$x_i$$ alone. Using the first-order multivariate Taylor expansion,

$$f(\mathbf{x} + \boldsymbol{\epsilon}) = f(\mathbf{x}) + \boldsymbol{\epsilon}^	op 
abla f(\mathbf{x}) + \mathcal{O}(\|\boldsymbol{\epsilon}\|^2)$$

one can show that the steepest-descent direction (up to second-order terms) is the negative gradient $$-
abla f(\mathbf{x})$$. Choosing a suitable learning rate $$\eta > 0$$ yields the multivariate gradient descent update rule:

$$\mathbf{x} \leftarrow \mathbf{x} - \eta 
abla f(\mathbf{x})$$

This directly generalizes the scalar update $$x \leftarrow x - \eta f'(x)$$ to vector-valued parameters.

Multivariate Gradient Descent

Dive into Deep Learning

To demonstrate multivariate gradient descent in practice, consider the two-dimensional objective function $$f(\mathbf{x}) = x_1^2 + 2x_2^2$$ with input $$\mathbf{x} = [x_1, x_2]^\top$$. Its gradient is $$\nabla f(\mathbf{x}) = [2x_1, 4x_2]^\top$$. Starting from the initial point $$[-5, -2]$$ and applying the update rule $$\mathbf{x} \leftarrow \mathbf{x} - \eta \nabla f(\mathbf{x})$$ with a learning rate of $$\eta = 0.1$$ for $$20$$ iterations, the trajectory of $$\mathbf{x}$$ converges steadily toward the minimum at $$[0, 0]$$. After $$20$$ steps the parameters reach approximately $$x_1 \approx -0.057646$$ and $$x_2 \approx -0.000073$$, confirming well-behaved but relatively slow convergence. The contour plot of $$f$$ shows elliptical level curves (elongated along $$x_1$$) because the curvature in the $$x_2$$ direction is twice that in $$x_1$$, which causes $$x_2$$ to converge faster than $$x_1$$.

Learn Before

Related

Learn After