https://towardsdatascience.com/how-to-debug-a-neural-network-with-gradient-checking-41deec0357a9

University of Michigan - Ann Arbor

This is the technique used to check that our implementation is correct. There are different formulas for gradient checking and one of those is two-sided form:
          $$\frac {J(\theta + \epsilon) - J(\theta - \epsilon)} {2\epsilon}$$
Common choice for $\epsilon$ is $10^{-7}$. You shouldn't use gradient checking for the whole training data as it can be slow.

Learn Before

Related