Deep learning earned its name since 2006, which is the third wave of deep learning. On 2006, Geoﬀrey Hinton showed that neural network can be efficiently trained with a strategy called greedy layer-wise pretraining. Researchers has been developing on deep learning vastly ever since.

Johns Hopkins University

Deep learning has a rich history that comes under different names. It develops with accelerated speed due to the increasing size of datasets and capacity of computer infrastructures in the recent decades. The complexity of tasks that deep learning can handle has increased.

Deep Learning History

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Deep learning is named cybernetics in 1940s - 1960s. Neural Networks are inspired by biological learning that attempts to simulate the functions of a human brain. Modern deep learning goes beyond biological perspectives and appeals to a more general principle: learning multiple level of composition. The earliest predecessors of neural network were linear models. Algorithms to learn the weights of these models were then developed, among which stochastic gradient descent is the most sophisticated and effective one. Neuroscience has become less of a source of inspiration due to the nature of complexity of human brains. Basic principles of how a brain works inspired the creation of convolutional neural network(CNN). However, in modern days, deep learning draws inspiration from many fields, including linear algebra, probability, information theory, and numerical optimization. Neuroscience is not a necessary field to study deep learning.

Deep Learning as Cybernetics

Deep learning attempts to solve problems that are intuitive for human beings by utilizing a hierarchical structure to capture concepts. Traditional approaches define formal and mathematical rules to solve problems. 

Where "AI Deep Learning" comes from

Deep learning is named connectionism in 1980s - 1990s. The wave of connectionism has risen, due the development of cognitive science. Distributed representation was developed to learn the high representation of objects, which greatly simplifies the deep learning algorithms. Back-propagation is another achievement at this time. However, AI research did not fulfill the unreasonable expectation from investors, the rise of Kernel Machines and graphical models soon took the stages and caused the decline of deep learning until 2007.

Deep Learning as Connectionism

Modern Deep Learning History

Deep learning is named cybernetics in 1940s - 1960s, connectionism in 1980s - 1990s and deep learning beginning 2006. 

"Deep Learning" Names in History

A large training dataset is necessary for the well performance of deep learning. During the past dacades, our computers are increasingly networked together, and thus it became relatively easy to obtain more information, and easy to feed the deep learning model appropriate amount of data to learn. Some big data sets that's common now could be MNIST, which contains about 10^5 datapoints.

Increasing training data sets on deep learning

Through the research of biological neurons, people found that the more connections each neuron has, the more intelligent the animal could be. With the development of hardwares such as GPU as well as the implementation of hidden layers, the size of neural network models has increased largely, and thus it is now possible to reach a better performance.

Learn Before

Related