If the gradient g is ill-conditioned, it will cost more time to train the model since in this situation the learning rate must shrink to cancel out the increase of gradient norm.

Ill Conditioning in Neural Network Training

Conditioning is used to describe the rate of change of a function as the input changes.
- The $\textbf{conditioning number}$ is used to describe how rapidly the function changes

Ohio State University, Columbus

Computations in finitie fields

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Underflow is a form of rounding error that occurs when a number near zero is rounded to zero.

Underflow

Overflow is a form of rounding error that occurs when a large number is rounded to ∞ or -∞.

Overflow

This function is used to predict the probabilities of values in a multinoulli distribution. This function must be stabilized against underflow and overflow.

Learn Before

Related

Learn After