A defining characteristic of deep learning research and practice is its strong reliance on empiricism, particularly when dealing with complex, nonconvex nonlinear optimization problems. Instead of requiring formal mathematical proofs of optimality or convergence before adopting an algorithm, practitioners prioritize rapid experimentation. They are willing to accept theoretically suboptimal solutions if those solutions yield superior practical accuracy, an approach that has significantly accelerated the development of practical deep learning tools.