Can have a statistical advantage when an apparently complicated structure can be compactly represented with a small number of parameters. Some traditional nondistributed learning algorithms generalize only due to the smoothness assumption, which states that if $u \approx v$, then the target function $f$ to be learned has the property that $f(u) \approx f(v)$ in general, which is useful, but suffers from the curse of dimensionality. 

Benefits of Distributed Representations

The curse of dimensionality states that as the number of dimensions (features) in the data increase, the number of categories of interest increases exponentially. This makes using machine learning on high dimensional data difficult.

For a data set with $d$ dimensions and $v$ values to be categorized for each dimension, $O(v^d)$ regions and examples are needed.


University of Connecticut

Deep learning is taking off because as the amount of data and computational power increases, deep learning algorithms perform better than traditional machine learning methods. 

You can improve performance by increasing the scale of the NN or giving it more data. Increasing the scale of the NN often places greater strain on technological systems. The more accessible route is taking advantage of the increase in available data.

“Why is deep learning taking off?” 

The inability for traditional algorithms to generalize complex problems was what inspired machine learning in the first place. Today, a similar situation has arose that complex problems with high dimensional inputs (high number of features) are not generalizable with most simple machine learning algorithms, not to mention require high computational power to solve. This is what drives the development of deep learning, as it will solve some problems better than machine learning can.

Challenges Motivating Deep Learning

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

https://machinelearning-blog.com/2017/11/03/erster-blogbeitrag/#:~:text=Getting%20a%20better%20accuracy%20with,or%20huge%20amounts%20of%20data.&text=The%20recent%20breakthroughs%20in%20the,use%20more%20and%20more%20data.

If the basic technical ideas behind Deep Learning are around for decades, why are they taking off today? 

https://towardsdatascience.com/how-do-you-know-you-have-enough-training-data-ad9b1fd679ee

Why deep learning is taking off source

https://medium.com/@shaoliang.jia/why-is-deep-learning-taking-off-8244175929e7

Learn Before

Related

Learn After