- **Pros**: The original study and practical experience from the industry shows that Bayesian optimization results in significantly higher accuracy compared to random search. 
- **Cons**: Like random search, results are not intuitive and difficult to improve on, even by trained operators.

University of Michigan - Ann Arbor

Bayesian optimization (described by Shahriari, et al.) trains the model with different hyperparameter values, and observes the function generated for the model by each set of parameter values repeatedly. Each time selecting hyperparameter values that are slightly different and can help plot the next relevant segment of the problem space. Like sampling methods in statistics, the algorithm ends up with a list of possible hyperparameter value sets and model functions, from which it predicts the optimal function across the entire problem set. 

Bayesian Optimization for Hyperparameter Tuning Methods in Deep Learning

According to a 2012 research study by James Bergstra and Yoshua Bengio, testing randomized values of hyperparameters is actually more effective than manual search or grid search. In other words, instead of testing systematically to cover “promising areas” of the problem space, it is preferable to test random values drawn from the entire problem space. 

Random Search for Hyperparameter Tuning Methods in Deep Learning

A helpful website that introduces neural networks:
https://missinglink.ai/guides/neural-network-concepts/

Neural Network Reference

Bayesian Optimization vs Random Search for Hyperparameter Tuning Methods in Deep Learning

- **Pros**: According to the study, provides higher accuracy with less training cycles, for problems with high dimensionality 
- **Cons**: Results are unintuitive, difficult to understand “why” hyperparameter values were chosen

Learn Before

Related