Tuning a Tv has many settings. There may be one knob for color, one for picture centering, and one for brightness. Each setting has a different knob, meaning it is easier to attain your desired screen by fixing each quality one at a time. 


Example of orthogonalization


When optimizing a classifier, we can optimize the values/settings of each of the hyperparameters separately from each other. Orthogonalization assumes that hyperparameters are independent of each other.

University of Michigan - Ann Arbor

In this step, we search for the most optimal values for (hyper)parameters that give the best classification model for the chosen evaluation criterion. For example, we can try a range of values for the "k" parameter in k-nearest neighbor classifier.

Classification Problem Optimization

- Grid search
- Randomly choosing
- Coarse to fine

Hyperparameters Tuning Methods in Deep Learning

https://www.coursera.org/learn/machine-learning-projects?specialization=deep-learning

Structuring Machine Learning Projects

Domingos, P. (2012). A few useful things to know about machine learning. Communications of the ACM, 55(10), 78-87. doi:10.1145/2347736.2347755

A few useful things to know about machine learning

Which one(s) are examples of "Optimization" in classification problems?

Orthogonalization

Grid Search may be very inefficient because it requires training the model for each of the combinations of values to calculate and compare the dev error.
A more efficient method is to take only random combinations from the grid and compare the dev error on those random combinations.

Randomly Tuning Hyperparameters in Deep Learning

Starting with parameter values far from each other and as we find regions of better performance, we zoom in those areas to find the optimal values for hyperparameters in those regions. Then, we find the most optimal values in that area and continue zooming in more details.

Coarse to Fine Tuning Hyperparameters in Deep Learning

Grid search is choosing the most optimal values for hyperparameters from the grid of all possible values, which may be very costly.

Learn Before

Related

Learn After