Grid search is choosing the most optimal values for hyperparameters from the grid of all possible values, which may be very costly.

University of Michigan - Ann Arbor

- Grid search
- Randomly choosing
- Coarse to fine

Hyperparameters Tuning Methods in Deep Learning

Grid Search may be very inefficient because it requires training the model for each of the combinations of values to calculate and compare the dev error.
A more efficient method is to take only random combinations from the grid and compare the dev error on those random combinations.

Randomly Tuning Hyperparameters in Deep Learning

Starting with parameter values far from each other and as we find regions of better performance, we zoom in those areas to find the optimal values for hyperparameters in those regions. Then, we find the most optimal values in that area and continue zooming in more details.

Coarse to Fine Tuning Hyperparameters in Deep Learning

Grid Search for Tuning Hyperparameters in Deep Learning

When optimizing a classifier, we can optimize the values/settings of each of the hyperparameters separately from each other. Orthogonalization assumes that hyperparameters are independent of each other.

Learn Before

Related