- **Pros**: The original study and practical experience from the industry shows that Bayesian optimization results in significantly higher accuracy compared to random search. 
- **Cons**: Like random search, results are not intuitive and difficult to improve on, even by trained operators.

Bayesian Optimization vs Random Search for Hyperparameter Tuning Methods in Deep Learning

- **Pros**: According to the study, provides higher accuracy with less training cycles, for problems with high dimensionality 
- **Cons**: Results are unintuitive, difficult to understand “why” hyperparameter values were chosen

Pros and Cons of Random Search for Hyperparameter Tuning Methods in Deep Learning

According to a 2012 research study by James Bergstra and Yoshua Bengio, testing randomized values of hyperparameters is actually more effective than manual search or grid search. In other words, instead of testing systematically to cover “promising areas” of the problem space, it is preferable to test random values drawn from the entire problem space. 

University of Michigan - Ann Arbor

 - Manual Hyperparameter Tuning
 - Grid Search
 - Random Search
 - Bayesian Optimization

Hyperparameter Tuning Methods in Deep Learning

A helpful website that introduces neural networks:
https://missinglink.ai/guides/neural-network-concepts/

Neural Network Reference

Bayesian optimization (described by Shahriari, et al.) trains the model with different hyperparameter values, and observes the function generated for the model by each set of parameter values repeatedly. Each time selecting hyperparameter values that are slightly different and can help plot the next relevant segment of the problem space. Like sampling methods in statistics, the algorithm ends up with a list of possible hyperparameter value sets and model functions, from which it predicts the optimal function across the entire problem set. 

Bayesian Optimization for Hyperparameter Tuning Methods in Deep Learning

Random Search for Hyperparameter Tuning Methods in Deep Learning

Grid search is slightly more sophisticated than manual tuning. It involves systematically testing multiple values of each hyperparameter, by automatically retraining the model for each value of the parameter. For example, you can perform a grid search for the optimal batch size by automatically training the model for batch sizes between 10-100 samples, in steps of 20. The model will run 5 times and the batch size selected will be the one which yields the highest accuracy.

Grid Search for Hyperparameter Tuning Methods in Deep Learning

Traditionally, hyperparameters were tuned manually by trial and error. This is still commonly done, and experienced operators can “guess” parameter values that will achieve remarkably high accuracy for deep learning models. However, there is a constant search for better, faster, and more automatic methods to optimize hyperparameters. 

Learn Before

Related

Learn After