Before the training phase begins for the CIFAR-10 ResNet-18 model, its parameters must be initialized appropriately. A standard technique used in this setup is Xavier initialization, which assigns the initial weights to maintain stable activation variance across the network layers.

Claude

Xavier initialization, named after its creators Glorot and Bengio (2010), is a standard technique designed to mitigate vanishing and exploding gradients by carefully setting the initial weights of a neural network layer. To balance the variance during both forward and backward propagation, it typically samples weights from a Gaussian distribution with a mean of $$0$$ and a variance of $$\sigma^2 = \frac{2}{n_	extrm{in} + n_	extrm{out}}$$, where $$n_	extrm{in}$$ and $$n_	extrm{out}$$ represent the number of inputs and outputs of the layer respectively. While the underlying assumption of linear activations is often violated in practice, this initialization method has proven highly effective.

Xavier Initialization

Dive into Deep Learning

Xavier initialization can also be adapted for sampling weights from a uniform distribution instead of a Gaussian one. A uniform distribution $$U(-a, a)$$ has a variance of $$\frac{a^2}{3}$$. By setting this equal to the Xavier variance condition $$\sigma^2 = \frac{2}{n_	extrm{in} + n_	extrm{out}}$$ and solving for $$a$$, we obtain $$a = \sqrt{\frac{6}{n_	extrm{in} + n_	extrm{out}}}$$. Therefore, the uniform version of Xavier initialization samples weights according to the distribution $$U\left(-\sqrt{\frac{6}{n_	extrm{in} + n_	extrm{out}}}, \sqrt{\frac{6}{n_	extrm{in} + n_	extrm{out}}}ight)$$.

Learn Before

Related