https://people.csail.mit.edu/hmobahi/pubs/aaai_2015.pdf

University of Michigan - Ann Arbor

The underlying premise:
- In high dimensional feature spaces, there is a very large likelihood of encountering multiple local minima. 
- This is compounded with the fact that the feature space isn’t well behaved i.e. an inevitable presence of plateaus, saddle points affecting the magnitude of the gradient along with the direction of the gradient not pointing towards the approximate region of lower cost.

Thus, being able to initialize parameters in a region smaller than the feature space itself and one that behaves well enough for the descent to lead to a reasonable solution is powerful. This is exactly what continuation methods try to achieve.

Continuation Methods

So, can continuation methods still be of use?
Yes. they do really well at:
-  reducing estimation variance (The intuitive effect of blurring applies here too) in the gradients, 
- almost eliminate flat regions and 
- improve the conditioning of the Hessian matrix thereby reducing instability.

The real utility of continuation methods

Curriculum learning is an approach in machine learning where tasks are sequentially structured so that their complexity gradually increases. This allows models to master simpler tasks first, which prepares them for more complex challenges as their skills develop. It is utilized as a method to mitigate the impact of sparse rewards in reinforcement learning by providing an organized path of learning.

Curriculum Learning

Continuation methods do not universally solve non-convex optimization problems. In complex non-convex feature spaces, applying the continuation method can generate a computationally intractable number of objective functions, meaning that NP-hard problems remain NP-hard. Furthermore, some non-convex problems remain non-convex regardless of the extent of blurring (e.g., $$J(\Theta)=-\Theta^{T}\Theta$$). Additionally, blurring the space can create a false convex structure, forcing the solution path toward inferior local minima. While local minima and saddle points are problematic, sparsity is often a more significant challenge in high-dimensional feature spaces.

Limitations of Continuation Methods for Non-Convex Problems

Continuation methods aim to minimize a complex cost function $$J(\Theta)$$ by constructing a sequence of increasingly difficult cost functions $$\left\{J^{(0)}, \dots, J^{(n)}\right\}$$, where $$J^{(0)}$$ is the most smoothed (easiest to optimize) and $$J^{(n)}=J(\Theta)$$ is the original function. Smoothing is often achieved by "blurring" the cost function via sampling, such as approximating it with $$J^{(i)}(\Theta)=\mathop{\mathbb{E}}_{\Theta'\sim\mathcal{N}(\Theta';\Theta,\sigma^{(i)2})}J(\Theta')$$. Optimization begins at $$J^{(0)}$$, where the blurred non-convex space resembles a convex one, providing an effective initial starting point for the subsequent function $$J^{(1)}$$. Iterating this process through $$J^{(n)}$$ typically yields a superior local minimum for the original optimization problem, though it does not guarantee finding the global minimum.

Learn Before

Related

Related