Rather than training the entire network at the same time, this technique involves training one layer at a time using a greedy approach. This is where the broader problem is broken down into smaller subproblems, and then the optimal solution to each subproblem is combined to make the solution to the larger problem. Sometimes this takes the form of training the input layers first and then moving through the network, and sometimes the layers near the input and output are trained and new hidden layers are added in the middle of the two.

Greedy Supervised Pretraining

This method of optimization involves applying algorithm concepts to training other algorithms, hence the "meta-" prefix. 

University of Virginia, Charlottesville

Deep Learning models require incredible computational resources to successfully train a model. As a result, optimizing the training process is a key step in developing an algorithm. One approach involves finding the parameters of a neural network that minimize its loss function. This is accomplished via gradient descent, a popular technique for minimizing loss function error. 

Optimization for Training Deep Models

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

The best strategy to improve optimization is by designing models that are easier to optimize. For example, linear paths or skip connections between layers reduce the length of the shortest path from the lower layer’s parameters to the output, and thus mitigate the vanishing gradient problem. Model innovations like the LSTM, rectiﬁed linear units and maxout units have all moved toward using linear functions to improve optimization. 

Designing Models to Aid Optimization

Machine learning acts indirectly, specifically, with loss fucntion having relationship with but not the same with the measurement P, which pure optimization always wants to optimize.

How does learning differ from pure optimization?

The training process based on minimizing this average training error is known
as empirical risk minimization.

Empirical Risk Minimization

Meta-Algorithm Approaches

Empirical risk is defined as the empirical average of the loss function. In this case, empirical means the performance was evaluated with real-life testing.

Empirical Risk

Surrogate loss function is a proxy having consistency or calibration with the original loss function, which is not comfortable for people to use derivatives or other direct method to optimize.

Surrogate Loss Function

Early Stopping means stopping earlier when the loss function doesn't have relatively obvious progress or with worse validation in one or several recent epochs.

Early Stopping

The underlying premise:
- In high dimensional feature spaces, there is a very large likelihood of encountering multiple local minima. 
- This is compounded with the fact that the feature space isn’t well behaved i.e. an inevitable presence of plateaus, saddle points affecting the magnitude of the gradient along with the direction of the gradient not pointing towards the approximate region of lower cost.

Thus, being able to initialize parameters in a region smaller than the feature space itself and one that behaves well enough for the descent to lead to a reasonable solution is powerful. This is exactly what continuation methods try to achieve.

Continuation Methods

Indexed notation is commonly used to represent a subsequence of elements, such as vectors. A sequence starting from index $m$ and ending at index $len$ is written as $x_m, x_{m+1}, \dots, x_{len-1}, x_{len}$. In the context of sequence models, this notation is frequently used for different components. For instance, a sequence of hidden states is often denoted as $h_m, h_{m+1}, \dots, h_{len}$, while a sequence of embeddings might be represented as $\mathbf{e}_m, \mathbf{e}_{m+1}, \dots, \mathbf{e}_{len}$.

Learn Before

Related

Learn After