The goal to minimize the cost function $J(\Theta)$ is achieved with the following adaptation. A set of cost functions {$J^{(0)},...,J^{(n)}$} with increasing difficulty are constructed such that $J^{(0)}$ is the easiest to optimize and $J^{(n)}=J(\Theta)$ has the highest level of difficulty.

What does “easier to optimize” mean?

• It means that it is relatively more well-behaved i.e. smoother over the same region.

This gets us a good initial start point for $J^{(i+1)}$. Continuing this would result in getting us very close to solving the $J(\Theta)$ optimization problem.

How do we get cost functions to behave well in the same region?

• By “blurring” out the cost function $J(\Theta)$ i.e. by approximating $J^{(i)}(\Theta)=\mathop{\mathbb{E}}_{\Theta^{’}\sim\mathcal{N}(\Theta_{‘};\Theta,\sigma^{(i)2})}J(\Theta^{‘})$ via sampling. The intended effect is that the non-convex problem starts to look like a convex one.

It is important to note that this method generally does not get us the global minima but does get us a superior local minima.