In each layer, before we put our input $z^{(i)}$ to the activation function, we first normalize it by Batch Norm. That is, we first calculate $\mu = \frac{1}{n}\sum_{i} z^{(i)}$ $\sigma^2=\frac{1}{n}\sum_{i}(z^{(i)}-\mu)^2$ $z_{norm}^{(i)} = \frac{z^{(i)}-\mu}{\sqrt{\sigma^2+\epsilon}}$ $\tilde{z}^{(i)}=\gamma^{(i)} z_{norm}^{(i)} + \beta^{(i)}$ We introduce $\gamma^{(i)}$ and $\beta^{(i)}$ because we don't want all the inputs to neurons in hidden layers to always have mean 0 and variance 1. $\Rightarrow a^{(i)} = g(\tilde{z}^{(i)})$, where g is some activation function. Recall in each layer, $z^{(i)}= W^{(i)} a^{(i-1)}+b^{(i)}$ When we calculate $\tilde{z}^{(i)}$, we first normalize $z^{(i)}$ by subtracting the mean. So the value of $b^{(i)}$ has no influence on the result. $\Rightarrow$ In each layer, we have three parameters $W^{(i)}, \gamma^{(i)}, \beta^{(i)}$. Frequently, we combine this with mini-batches; i.e., we train $W^{(i)}$, $\gamma^{(i)}$, $\beta^{(i)}$ with respect to $X^{\{1\}}, ..., X^{\{n\}}$, where $X^{\{i\}}$ is $i^{th}$ batch of the training data.