This problem arises from the fact that fixing a local problem does not necessarily provide a global solution to a function. 

One of the most popular solutions to solving the disparity between the local and global structure is to design a way to finding good initial points for problems with difficult global solutions rather than coming up with an algorithm that uses non-local moves.  

In the example below, the cost function contains only asymptotes toward low values, not minima. The main cause of diﬃculty in this case is being initialized on the wrong side of the “mountain” and not being able to traverse it. The simple and most optimal solution is to move the starting point to the right side of the “mountain”. 



Nova Southeastern University

- **Local optima**: it's actually unlikely to get stuck in local optima.
- **Cliffs**: on the face of an extremely steep cliﬀ structure, the
gradient update step can move the parameters extremely far
- **Inexact Gradients**: sometimes approximation is needed for gradients
- **Plateaus**: low cost function slope (close to flat) makes learning slow.

Challenges with Deep Learning Optimizer Algorithms

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Most optimization techniques are constructed with the idea that one has access to the exact gradient; however, we often find ourselves making rough approximations or even biased estimates for the gradient, especially when the objective function we want to minimize is intractable the only option is to make an approximation. Thankfully, many neural network algorithms have some method in their design to accommodate for these imperfections.      

Learn Before

Related