https://datascience.stackexchange.com/questions/24534/does-gradient-descent-always-converge-to-an-optimum

University of Michigan - Ann Arbor

Gradient Descent is an algorithm which is designed to find the optimal points, but these optimal points are not necessarily global. And yes if it happens that it diverges from a local location it may converge to another optimal point but its probability is not too much.

Consider the following "recliner chair" type of function(image below).

Obviously, this can be constructed so that there is a range in the middle where the gradient is the 0 vector, casuing the fail to find global optima. 


Why Gradient descent might fail?

Reference to: Why Gradient descent might fail?



If failure is defined as a model becoming unable to escape a non-global local minima through training, then it is almost guaranteed that this will occur with a non-trivial and finite training set. This can be attributed to inherent $\textbf{model identifiability}$ issues which may allow uncountably infinite local minima to exist.

In practice, however, experts now believe that most relevant local minima are already close to the global minimum when given sufficiently large training sets. Although this is still an active area of research.

Learn Before

Related