A global minimum is the point at which an objective function $$f(x)$$ achieves its absolute lowest value across its entire domain. Unlike a local minimum, which only represents the lowest value within a limited vicinity, the global minimum represents the true, optimal numerical solution for the overall optimization problem.

Global Minimum

To visualize the distinction between minima, consider the objective function $$f(x) = x \cdot \cos(\pi x)$$ defined over the domain $$-1.0 \leq x \leq 2.0$$. By evaluating and plotting this function, one can approximate both its local minimum—a point lowest only in its immediate vicinity—and its global minimum, which represents the lowest value across the entire specified interval.

Local and Global Minima Approximation

Because gradients approach zero at a local minimum, an optimization algorithm can become trapped. However, introducing some degree of noise can knock the model's parameters out of these sub-optimal valleys. One of the beneficial properties of minibatch stochastic gradient descent is that the natural statistical variation of gradients evaluated over different minibatches inherently provides this necessary noise, successfully dislodging the parameters from local minima.

Escaping Local Minima

For a function whose input is a multi-dimensional vector and output is a scalar, a zero-gradient position represents a local minimum if all the eigenvalues of the function's Hessian matrix at that position are positive.

Local Minimum via Hessian Eigenvalues

Optimization in deep learning is fraught with challenges, making it extremely difficult to consistently find the true global minimum of an objective function. However, finding the absolute best solution is rarely necessary in practice. Fortunately, there exists a robust range of optimization algorithms that are reliable and easy to use. The local optima or approximate solutions discovered by these algorithms are typically highly effective and perfectly sufficient for training successful models.

Acceptability of Suboptimal Solutions in Deep Learning

For nonconvex objective functions, gradient descent can converge to a local minimum rather than the global minimum, and the particular local minimum reached depends on both the learning rate and the problem's conditioning. As an illustration, consider the function $$f(x) = x \cdot \cos(cx)$$ for a constant $$c$$, which possesses infinitely many local minima due to its oscillatory structure. When gradient descent is applied with an unrealistically high learning rate, the algorithm takes large steps that skip over better-quality minima and settles into a poor local minimum. This demonstrates that the learning rate not only affects convergence speed but also influences which solution gradient descent ultimately finds on nonconvex landscapes.

Gradient Descent on a Nonconvex Function with Local Minima

For a given objective function $$f(x)$$, a local minimum occurs at a point $$x$$ if the function's value $$f(x)$$ is smaller than its values at all other points within its immediate vicinity. In deep learning, objective functions usually possess many of these local optima. As an optimization algorithm nears a local minimum, the gradient approaches zero, which can cause the final numerical solution to only minimize the function locally rather than discovering the global minimum.

Claude

University of Michigan - Ann Arbor

University of Toledo

Points where the derivative of a function is 0 are known as critical points, or stationary points.

The derivative also has to be zero according to every possible directional derivative at that point in higher dimensional functions.

$$\triangledown f = 0$$

Critical Points

While deep learning optimization faces many hurdles, some of the most vexing challenges encountered when minimizing complex objective functions include getting trapped in local minima, stalling at saddle points, and suffering from vanishing gradients.

Vexing Optimization Challenges in Deep Learning

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Dive into Deep Learning

A local maximum is a critical points which function value is higher than at all neighboringpoints.

Local Maximum

Local Minimum

A saddle point is a location where all gradients of a function vanish, but it is neither a global nor a local minimum. For example, the function $$f(x) = x^3$$ has a saddle point at $$x = 0$$, where its first and second derivatives are zero. Optimization algorithms might stall at these points because the gradient is zero, even though they do not represent true minima.

Saddle Point

In a neural network with many time steps or layers, a gradient at the early layer is the product of all the terms from the later layers, which leads to an inherently unstable situation. Especially when the value of gradient has become so small, it no longer updates properly or is vanished eventually. Exploding gradient can be considered as the opposite of vanishing process. The updated weights using gradient descent become so large that they cause the whole network to become unstable, which leads to numerical overflow.

Learn Before

Related

Learn After