From 2006 to 2012, it was widely believed that feedforward networks would not perform well unless they were assisted by other models, such as probabilistic models. We now know that with the right resources and engineering practices, feedforward networks perform very well. 

Today, gradient-based learning in feedforward networks is used as a tool to develop probabilistic models, such as the variational autoencoder and generative adversarial networks. In 2006, the community used unsupervised learning to support supervised learning, and now, ironically, it is more common to use supervised learning to support unsupervised learning.



Deep Feedfoward Networks in current times.

-The book Parallel Distributed Processing (published in the 1980s) presented the results of some of the ﬁrst successful experiments with back-propagation. 
-Following the success of Parallel Distributed Processing, back-propagation had a wave of success and popularity that peaked in the early 1990s. From then on, other machine learning teachings and techniques gained more popularity until the modern deep learning “renaissance” in 2006. 


Nova Southeastern University

-The chain rule that underlies the back-propagation algorithm (one of the common algorithms used to train neural networks) was invented in the seventeenth century.

-While algebra and advanced mathematics have always been implemented to solve computer optimization problems, gradient descent was not introduced as a technique for iteratively approximating the solution to optimization problems until the nineteenth century.

-Non-linear functions could only be properly studied once a multilayer perceptron and a means of computing the gradient were developed. One of the biggest advancements for this development occurred once eﬃcient applications of the chain rule based on dynamic programming were implemented between the 1960s and 70s. 


Deep Feedfoward Network as time passed.

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Learn Before

Related

Learn After