Grid Search may be very inefficient because it requires training the model for each of the combinations of values to calculate and compare the dev error.
A more efficient method is to take only random combinations from the grid and compare the dev error on those random combinations.

Randomly Tuning Hyperparameters in Deep Learning

Starting with parameter values far from each other and as we find regions of better performance, we zoom in those areas to find the optimal values for hyperparameters in those regions. Then, we find the most optimal values in that area and continue zooming in more details.

Coarse to Fine Tuning Hyperparameters in Deep Learning

Grid search is choosing the most optimal values for hyperparameters from the grid of all possible values, which may be very costly.

Grid Search for Tuning Hyperparameters in Deep Learning

When optimizing a classifier, we can optimize the values/settings of each of the hyperparameters separately from each other. Orthogonalization assumes that hyperparameters are independent of each other.

Orthogonalization

- Grid search
- Randomly choosing
- Coarse to fine

University of Michigan - Ann Arbor

Hyperparameters related to neural network structure:
- Number of hidden layers (Depth)
- Number of hidden units (Width)
- Dropout method
- Activation function for each layer
- Weights Initialization

Hyperparameters related to training algorithm:
- Learning rate $$\alpha$$
- Momentum parameter $$\beta \sim 0.9$$
- $$\beta_1 \sim 0.9, \beta_2 \sim 0.999, \epsilon \sim 10^{-8}$$
- Number of Gradient descent iterations
- Mini-batch size
- Optimizer algorithm
- Learning rate decay
- Regularization rate $$\lambda$$

List of Common Hyperparameters in Deep Learning

For a feedforward neural network, the depth of the network is the number of hidden layers plus one (as the output layer is also parameterized). The width of the network is the dimensionality of its hidden layers.

The main architectural considerations for designing a neural network are choosing the depth of the network and the width of each layer.


Depth and Width for Neural Networks

Dropout means what percentage of neurons should be randomly “killed” during each epoch to prevent overfitting.

Dropout

Neural network learning rate means how fast the backpropagation algorithm performs gradient descent. A larger learning rate makes the network train faster but might result in missing the minimum of the loss function.

Neural Network Learning Rate

When you try to train a huge amount of data with limited computer memory, you can separate the whole training set to several batches that can fit into your computer memory. Then you feed these batches to your model one by one. After feeding all batches once, you complete one epoch. To successfully train your model, you need multiple epochs.

Epochs in Machine Learning

In a neural network, numeric data points, called inputs, are fed into the neurons in the input layer. Each neuron has a weight, and multiplying the input number with the weight gives the output of the neuron, which is transferred to the next layer.
The activation function is a mathematical “gate” in between the input feeding the current neuron and its output going to the next layer. It can be as simple as a step function that turns the neuron output on and off, depending on a rule or threshold. Or it can be a transformation that maps the input signals into output signals that are needed for the neural network to function.
Activation functions are mathematical equations that determine the output of a neural network. The function is attached to each neuron in the network, and determines whether it should be activated (“fired”) or not, based on whether each neuron’s input is relevant for the model’s prediction. Activation functions also help normalize the output of each neuron to a range between 1 and 0 or between -1 and 1. When used in a multilayer neural network, activation functions can be different for different layers. 

Activation Functions in Neural Networks

When a neural network trains, it uses an algorithm to determine the optimal weights for the model, called an optimizer. 
There are several optimizer algorithms, such as:
- Gradient descent
- Mini-batch gradient descent
- Gradient descent with momentum
- RMSprop
- Adam optimization algorithm
- Nesterov momentum
- AdaGrad

Deep Learning Optimizer Algorithms

Initial weights are applied to all the neurons.
It is necessary to set initial weights for the first forward pass. Two basic options are to set weights to zero or to randomize them. However, this can result in a vanishing or exploding gradient, which will make it difficult to train the model. To mitigate this problem, you can use a heuristic (a formula tied to the number of neuron layers) to determine the weights. A common heuristic used for the Tanh activation is called Xavier initialization.

On top of that if the architecture being used has already been trained by someone else (such as ImageNet) the node values can be initialized to those values. This is known as transfer learning and it can be very useful to substantially increase training speed and model accuracy.

Deep Learning Weight Initialization

Hyperparameters Tuning Methods in Deep Learning

The regularization constant, $$\lambda$$, is a nonnegative hyperparameter that characterizes the trade-off between standard prediction loss and a regularization penalty. It is typically fit using validation data. When $$\lambda = 0$$, the original loss function is recovered, while larger values of $$\lambda$$ constrain the model's weights more considerably. For example, in weight decay ($$\ell_2$$ regularization), it modifies the objective to $$L(\mathbf{w}, b) + \frac{\lambda}{2} \|\mathbf{w}\|^2$$, where the penalty term is conventionally divided by 2 so that the constant cancels out gracefully when taking the derivative.

Regularization Constant

Batch normalization is a technique designed to accelerate and stabilize the training of deep neural networks. Mechanistically, it centers and rescales the intermediate layer activations back to a controlled mean and variance, preventing their distributions from diverging across layers and over time. By keeping these intermediate values on a comparable scale, batch normalization enables the use of more aggressive learning rates. The technique was originally motivated by the concept of covariate shift applied to internal layers, but the hypothesis that it works by reducing this so-called internal covariate shift has since been challenged and does not appear to be a valid explanation for its effectiveness. Although intuitively thought to make the optimization landscape smoother, the precise mechanism by which batch normalization aids training remains an open research question. Despite this theoretical uncertainty, batch normalization has proven indispensable in practice, being applied in nearly all deployed image classifiers and earning the original paper tens of thousands of citations.

Learn Before

Related

Learn After