Xavier initialization can also be adapted for sampling weights from a uniform distribution instead of a Gaussian one. A uniform distribution $$U(-a, a)$$ has a variance of $$\frac{a^2}{3}$$. By setting this equal to the Xavier variance condition $$\sigma^2 = \frac{2}{n_	extrm{in} + n_	extrm{out}}$$ and solving for $$a$$, we obtain $$a = \sqrt{\frac{6}{n_	extrm{in} + n_	extrm{out}}}$$. Therefore, the uniform version of Xavier initialization samples weights according to the distribution $$U\left(-\sqrt{\frac{6}{n_	extrm{in} + n_	extrm{out}}}, \sqrt{\frac{6}{n_	extrm{in} + n_	extrm{out}}}ight)$$.

Michigan State University

Claude

• The main steps for building a Neural Network are:

     – Define the model structure (such as number of input features and outputs)

     – Initialize the model’s weights and biases.

     – Loop.

           ∗ Calculate current loss (forward propagation)

           ∗ Calculate current gradient (backward propagation)

           ∗ Update parameters (gradient descent)

• Preprocessing the dataset is important.

• Tuning the learning rate (which is an example of a “hyperparameter”) can make a big difference to the algorithm.

Main Step to build a NN

Xavier initialization, named after its creators Glorot and Bengio (2010), is a standard technique designed to mitigate vanishing and exploding gradients by carefully setting the initial weights of a neural network layer. To balance the variance during both forward and backward propagation, it typically samples weights from a Gaussian distribution with a mean of $$0$$ and a variance of $$\sigma^2 = \frac{2}{n_	extrm{in} + n_	extrm{out}}$$, where $$n_	extrm{in}$$ and $$n_	extrm{out}$$ represent the number of inputs and outputs of the layer respectively. While the underlying assumption of linear activations is often violated in practice, this initialization method has proven highly effective.

Xavier Initialization

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Dive into Deep Learning

For a neural network,  you shouldn't initialize weights to parameters to all zero.
Why is that?
Let's take an example:
you have two input features, x1 and x2, and two hidden units. So the matrix of this layer's parameter W will be a 2 by 2 matrix. 
If we initialize W to be full of 0, as it's shown in the image, a1=a2, and dz1=dz2 and these two neurons will compute exactly the same thing. And no matter how many layers you have, these two neurons will have same outputs.

To solve this problem, we can initialize our parameters by using 
np.random.randint((m,n))*0.01
multiple by 0.01 is to initlize a small number.



Random Initialization

A set number of nodes are initialized to nonzero values, while the rest are set to zero. This may result in a longer time to shrink overly large values, but can increase diversity among the units at initialization time,

Learn Before

Related