Bayesian optimization (described by Shahriari, et al.) trains the model with different hyperparameter values, and observes the function generated for the model by each set of parameter values repeatedly. Each time selecting hyperparameter values that are slightly different and can help plot the next relevant segment of the problem space. Like sampling methods in statistics, the algorithm ends up with a list of possible hyperparameter value sets and model functions, from which it predicts the optimal function across the entire problem set. 

Bayesian Optimization for Hyperparameter Tuning Methods in Deep Learning

According to a 2012 research study by James Bergstra and Yoshua Bengio, testing randomized values of hyperparameters is actually more effective than manual search or grid search. In other words, instead of testing systematically to cover “promising areas” of the problem space, it is preferable to test random values drawn from the entire problem space. 

Random Search for Hyperparameter Tuning Methods in Deep Learning

Grid search is slightly more sophisticated than manual tuning. It involves systematically testing multiple values of each hyperparameter, by automatically retraining the model for each value of the parameter. For example, you can perform a grid search for the optimal batch size by automatically training the model for batch sizes between 10-100 samples, in steps of 20. The model will run 5 times and the batch size selected will be the one which yields the highest accuracy.

Grid Search for Hyperparameter Tuning Methods in Deep Learning

Traditionally, hyperparameters were tuned manually by trial and error. This is still commonly done, and experienced operators can “guess” parameter values that will achieve remarkably high accuracy for deep learning models. However, there is a constant search for better, faster, and more automatic methods to optimize hyperparameters. 

Manual Hyperparameter Tuning for Hyperparameter Tuning Methods in Deep Learning

 - Manual Hyperparameter Tuning
 - Grid Search
 - Random Search
 - Bayesian Optimization

University of Michigan - Ann Arbor

Tuning or optimizing hyperparameters involves finding the values of each hyperparameter which will help the model provide the most accurate predictions.

Learn Before

Related

Learn After