Deep learning is named cybernetics in 1940s - 1960s. Neural Networks are inspired by biological learning that attempts to simulate the functions of a human brain. Modern deep learning goes beyond biological perspectives and appeals to a more general principle: learning multiple level of composition. The earliest predecessors of neural network were linear models. Algorithms to learn the weights of these models were then developed, among which stochastic gradient descent is the most sophisticated and effective one. Neuroscience has become less of a source of inspiration due to the nature of complexity of human brains. Basic principles of how a brain works inspired the creation of convolutional neural network(CNN). However, in modern days, deep learning draws inspiration from many fields, including linear algebra, probability, information theory, and numerical optimization. Neuroscience is not a necessary field to study deep learning.