If the gradient g is ill-conditioned, it will cost more time to train the model since in this situation the learning rate must shrink to cancel out the increase of gradient norm.

Ill Conditioning in Neural Network Training

Conditioning refers to how rapidly the output of a function changes in response to small changes in its input. This sensitivity is quantified by the $$\textbf{condition number}$$, which measures the rate of change of the function as the input varies.

Ohio State University, Columbus

Google

Computations in finitie fields

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Underflow is a form of rounding error that occurs when a number near zero is rounded to zero.

Underflow

Overflow is a form of rounding error that occurs when a large number is rounded to ∞ or -∞.

Overflow

This function is used to predict the probabilities of values in a multinoulli distribution. This function must be stabilized against underflow and overflow.

Learn Before

Related

Learn After