When randomly choosing values for hyperparameters to find the most optimal ones, sometimes, it's better to choose the random numbers based on specific distributions. For example, instead of choosing random numbers from a uniform distribution, it may be more reasonable to choose from a normal or logarithmic distribution. For choosing a value for:
- $0.0001 < \alpha < 1$, it's more reasonable to only try powers of 10 like $10^{-4}, ..., 10^0$.
- $0.9 < \beta < 0.9999$, it's more reasonable to only try numbers like 0.9, 0.99, 0.999, ..., 0.99999 because comparing 0.9000 and 0.9005, the latter results in ~10 more samples. On the other hand, 0.999 results in ~1000 samples but 0.9995 results in ~2000 samples.

Appropriate Scale for Randomly Tuning Hyperparameters in Deep Learning

Grid Search may be very inefficient because it requires training the model for each of the combinations of values to calculate and compare the dev error.
A more efficient method is to take only random combinations from the grid and compare the dev error on those random combinations.

University of Michigan - Ann Arbor

- Grid search
- Randomly choosing
- Coarse to fine

Hyperparameters Tuning Methods in Deep Learning

Randomly Tuning Hyperparameters in Deep Learning

Starting with parameter values far from each other and as we find regions of better performance, we zoom in those areas to find the optimal values for hyperparameters in those regions. Then, we find the most optimal values in that area and continue zooming in more details.

Coarse to Fine Tuning Hyperparameters in Deep Learning

Grid search is choosing the most optimal values for hyperparameters from the grid of all possible values, which may be very costly.

Grid Search for Tuning Hyperparameters in Deep Learning

When optimizing a classifier, we can optimize the values/settings of each of the hyperparameters separately from each other. Orthogonalization assumes that hyperparameters are independent of each other.

Learn Before

Related

Learn After