For Newton’s method, saddle points clearly constitute a problem. Gradient descent is designed to move “downhill” and is not explicitly designed to seek acritical point. Newton’s method, however, is designed to solve for a point where the gradient is zero. Without appropriate modiﬁcation, it can jump to a saddle point.

Newton's Method and Saddle Points

A saddle point is a location where all gradients of a function vanish, but it is neither a global nor a local minimum. For example, the function $$f(x) = x^3$$ has a saddle point at $$x = 0$$, where its first and second derivatives are zero. Optimization algorithms might stall at these points because the gradient is zero, even though they do not represent true minima.

Claude

University of Michigan - Ann Arbor

Points where the derivative of a function is 0 are known as critical points, or stationary points.

The derivative also has to be zero according to every possible directional derivative at that point in higher dimensional functions.

$$\triangledown f = 0$$

Critical Points

While deep learning optimization faces many hurdles, some of the most vexing challenges encountered when minimizing complex objective functions include getting trapped in local minima, stalling at saddle points, and suffering from vanishing gradients.

Vexing Optimization Challenges in Deep Learning

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Dive into Deep Learning

A local maximum is a critical points which function value is higher than at all neighboringpoints.

Local Maximum

For a given objective function $$f(x)$$, a local minimum occurs at a point $$x$$ if the function's value $$f(x)$$ is smaller than its values at all other points within its immediate vicinity. In deep learning, objective functions usually possess many of these local optima. As an optimization algorithm nears a local minimum, the gradient approaches zero, which can cause the final numerical solution to only minimize the function locally rather than discovering the global minimum.

Local Minimum

Saddle Point

In a neural network with many time steps or layers, a gradient at the early layer is the product of all the terms from the later layers, which leads to an inherently unstable situation. Especially when the value of gradient has become so small, it no longer updates properly or is vanished eventually. Exploding gradient can be considered as the opposite of vanishing process. The updated weights using gradient descent become so large that they cause the whole network to become unstable, which leads to numerical overflow.

Learn Before

Related

Learn After