 - Manual Hyperparameter Tuning
 - Grid Search
 - Random Search
 - Bayesian Optimization

Hyperparameter Tuning Methods in Deep Learning

How can we tell which parameters Θ to leave in the model (known as "model selection")?

Tuning or optimizing hyperparameters involves finding the values of each hyperparameter which will help the model provide the most accurate predictions.

University of Michigan - Ann Arbor

A hyperparameter is a setting that affects the structure or operation of the neural network. In real deep learning projects, tuning hyperparameters is the primary way to build a network that provides accurate predictions for a certain problem. 

Hyperparameters of Feedforward Neural Network

A helpful website that introduces neural networks:
https://missinglink.ai/guides/neural-network-concepts/

Neural Network Reference

One reason that people manipulate hyperparameters is to improve the bias and variance that a given model contains. Bias is the difference between the average prediction of our model and the correct value which we are trying to predict. Models with high bias pay little attention to the training data and oversimplify the model, which will lead to high errors on training and test sets. Variance is the variability of model prediction for a given data point or a value which tells us spread of our data. High Variance models do not generalize based on previous data much, if at all, and thus while it may perform well on training data it will not on test sets. Briefly, the Bias vs Variance scale is a scale between over- and under-fitting of a model. By manipulating various hyperparameters, you can change between over- and under-fitting to get the best predictions

Bias and Variance in Deep Learning

Hyperparameter Tuning in Deep Learning

Hyperparameters related to neural network structure:
- Number of hidden layers (Depth)
- Number of hidden units (Width)
- Dropout method
- Activation function for each layer
- Weights Initialization

Hyperparameters related to training algorithm:
- Learning rate $\alpha$
- Momentum parameter $\beta \sim 0.9$
- $\beta_1 \sim 0.9, \beta_2 \sim 0.999, \epsilon \sim 10^{-8}$
- Number of Gradient descent iterations
- Mini-batch size 
- Optimizer algorithm
- Learning rate decay
- Regularization rate $\lambda$ 

Learn Before

Related

Learn After