Tuning a Tv has many settings. There may be one knob for color, one for picture centering, and one for brightness. Each setting has a different knob, meaning it is easier to attain your desired screen by fixing each quality one at a time. 


Example of orthogonalization


When optimizing a classifier, we can optimize the values/settings of each of the hyperparameters separately from each other. Orthogonalization assumes that hyperparameters are independent of each other.

University of Michigan - Ann Arbor

In this step, we search for the most optimal values for (hyper)parameters that give the best classification model for the chosen evaluation criterion. For example, we can try a range of values for the "k" parameter in k-nearest neighbor classifier.

Classification Problem Optimization

- Grid search
- Randomly choosing
- Coarse to fine

Hyperparameters Tuning Methods in Deep Learning

https://www.coursera.org/learn/machine-learning-projects?specialization=deep-learning

Structuring Machine Learning Projects

Domingos, P. (2012). A few useful things to know about machine learning. Communications of the ACM, 55(10), 78-87. doi:10.1145/2347736.2347755

A few useful things to know about machine learning

Which one(s) are examples of "Optimization" in classification problems?

Grid Search may be very inefficient because it requires training the model for each of the combinations of values to calculate and compare the dev error.
A more efficient method is to take only random combinations from the grid and compare the dev error on those random combinations.

Randomly Tuning Hyperparameters in Deep Learning

Coarse-to-fine tuning is a hyperparameter search strategy that starts by sampling hyperparameter values spread widely across the search space. After training and evaluating models at these coarse points, the search zooms into the smaller region(s) where performance was best and samples more densely within that region to refine the values. This zoom-in step can be repeated, progressively narrowing the search until the most optimal hyperparameter values are found.

Coarse to Fine Tuning Hyperparameters in Deep Learning

Grid search is choosing the most optimal values for hyperparameters from the grid of all possible values, which may be very costly.

Learn Before

Related

Learn After

Learn After

Related