For a given objective function $$f(x)$$, a local minimum occurs at a point $$x$$ if the function's value $$f(x)$$ is smaller than its values at all other points within its immediate vicinity. In deep learning, objective functions usually possess many of these local optima. As an optimization algorithm nears a local minimum, the gradient approaches zero, which can cause the final numerical solution to only minimize the function locally rather than discovering the global minimum.

Local Minimum

In a neural network with many time steps or layers, a gradient at the early layer is the product of all the terms from the later layers, which leads to an inherently unstable situation. Especially when the value of gradient has become so small, it no longer updates properly or is vanished eventually. Exploding gradient can be considered as the opposite of vanishing process. The updated weights using gradient descent become so large that they cause the whole network to become unstable, which leads to numerical overflow.

Vanishing/exploding gradient

A saddle point is a location where all gradients of a function vanish, but it is neither a global nor a local minimum. For example, the function $$f(x) = x^3$$ has a saddle point at $$x = 0$$, where its first and second derivatives are zero. Optimization algorithms might stall at these points because the gradient is zero, even though they do not represent true minima.

Saddle Point

While deep learning optimization faces many hurdles, some of the most vexing challenges encountered when minimizing complex objective functions include getting trapped in local minima, stalling at saddle points, and suffering from vanishing gradients.

Claude

- **Local optima**: it's actually unlikely to get stuck in local optima.
- **Cliffs**: on the face of an extremely steep cliﬀ structure, the
gradient update step can move the parameters extremely far
- **Inexact Gradients**: sometimes approximation is needed for gradients
- **Plateaus**: low cost function slope (close to flat) makes learning slow.

Challenges with Deep Learning Optimizer Algorithms

Dive into Deep Learning

Most optimization techniques are constructed with the idea that one has access to the exact gradient; however, we often find ourselves making rough approximations or even biased estimates for the gradient, especially when the objective function we want to minimize is intractable the only option is to make an approximation. Thankfully, many neural network algorithms have some method in their design to accommodate for these imperfections.      

Inexact Gradients

This problem arises from the fact that fixing a local problem does not necessarily provide a global solution to a function. 

One of the most popular solutions to solving the disparity between the local and global structure is to design a way to finding good initial points for problems with difficult global solutions rather than coming up with an algorithm that uses non-local moves.  

In the example below, the cost function contains only asymptotes toward low values, not minima. The main cause of diﬃculty in this case is being initialized on the wrong side of the “mountain” and not being able to traverse it. The simple and most optimal solution is to move the starting point to the right side of the “mountain”. 



Learn Before

Related

Learn After