The universal approximation theorem (Horniket al., 1989; Cybenko, 1989) states that a feedforward network with a linear output layer and at least one hidden layer with any “squashing” activation function (such as the logistic sigmoid activation function) can approximate any Borel measurable function from one ﬁnite-dimensional space to another with any desired nonzero amount of error, provided that the network is given enough hidden units. The derivatives of the feedforward network can also approximate the derivatives of the function arbitrarily well.

It means that regardless of what function we are trying to learn, we know that a large feedforward network will be able to represent this function. 

Universal Approximation Theorem

The illustration below shows the relationship between the fields of Artificial Intelligence. Deep learning is a subset of machine learning. 

Refernce : https://lupinepublishers.com/material-science-journal/pdf/MAMS.MS.ID.000138.pdf

Set relationship of Artificial Intelligence

The possibility of Deep Learning is enormous, but it also has a limitation which we need to understand for better utilization. 

a. The systems need 10,000+ examples to learn a concept
like cows. Humans only need a handful of examples.

b. Deep Learning cannot explain how the systems got an
answer.

c. Deep Learning lacks common sense. This makes the
systems fragile and when errors are made, the errors can be
very large.


Limitation of Deep Learning

- It requires very large amount of data in order to perform better than other techniques.

- It is extremely expensive to train due to complex data models. Moreover deep learning requires expensive GPUs and hundreds of machines. This increases cost to the users.

- There is no standard theory to guide you in selecting right deep learning tools as it requires knowledge of topology, training method and other parameters. As a result it is difficult to be adopted by less skilled people.

- It is not easy to comprehend output based on mere learning and requires classifiers to do so. Convolutional neural network based algorithms perform such tasks.

Drawbacks or disadvantages of Deep Learning

In this node, we describe how to use deep learning to solve applications in computer vision, speech recognition, natural language processing, and other areas of commercial interest. 

Applications of Deep Learning

One of the most significant advantages of deep learning is its ability to replace the labor-intensive process of manual feature engineering. In traditional machine learning, domain experts crafted specific algorithms to transform raw perceptual data into feature vectors. Deep learning replaces these domain-specific preprocessing steps with automatically tuned filters that are learned jointly from the data, yielding superior accuracy and providing a unified approach across diverse fields.

Automated Feature Engineering

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

University of Michigan - Ann Arbor

- Deep Learning
- Deep learning and application in cancer detection
- Deep learning in a nutshell
- Top 15 Deep Learning applications that will rule the world in 2018 and beyond
- Delve into ReLU
- The ABCs of Approximate Bayes Computation
- Convolutional Recurrent Neural Networks For Music Classification.

Deep Learning (in Machine Learning) References

Deep Learning

https://ai.googleblog.com/2017/03/assisting-pathologists-in-detecting.html

Deep learning and application in cancer detection

https://developer.nvidia.com/blog/deep-learning-nutshell-core-concepts/

Deep learning in a nutshell

https://medium.com/breathe-publication/top-15-deep-learning-applications-that-will-rule-the-world-in-2018-and-beyond-7c6130c43b01

Top 15 Deep Learning applications that will rule the world in 2018 and beyond

https://www.kaggle.com/dansbecker/rectified-linear-units-relu-in-deep-learning

Delve into ReLU

Leyshon, T. (2021, March 22). The abcs of approximate bayesian computation. Medium. https://towardsdatascience.com/the-abcs-of-approximate-bayesian-computation-bfe11b8ca341. 

The ABCs of Approximate Bayesian Computation

Choi, K., Fazekas, G., Sandler, M., & Cho, K. (2017, March). Convolutional recurrent neural networks for music classification. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 2392-2396). IEEE.

CONVOLUTIONAL RECURRENT NEURAL NETWORKS FOR MUSIC CLASSIFICATION

Ruder, S. (n.d.). An Overview of Multi-Task Learning in Deep Neural Networks *. Retrieved May 22, 2022, from https://arxiv-download.xixiaoyao.cn/pdf/1706.05098.pdf

‌

Multi-Task Learning in Deep Neural Networks

Wang, Y., Yao, Q., Kwok, J. T., & Ni, L. M. (2020). Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3), 1-34.

Generalizing from a Few Examples: A Survey on Few-Shot Learning

Minaee, Shervin et al. Deep Learning Based Text Classification: A Comprehensive Review. Retrieved May 22, 2022, from https://arxiv.org/pdf/2004.03705.pdf

Deep Learning Based Text Classification: A Comprehensive Review

Zhang, Z., Cui, P., & Zhu, W. (2020). Deep learning on graphs: A survey. IEEE Transactions on Knowledge and Data Engineering.

Deep Learning on Graphs: A Survey

Han, Y., Huang, G., Song, S., Yang, L., Wang, H., & Wang, Y. (2021). Dynamic Neural Networks: A Survey. IEEE transactions on pattern analysis and machine intelligence, PP.
https://arxiv.org/abs/2102.04906

Dynamic Neural Networks: A Survey

Bayer, Marcus et al. A Survey on Data Augmentation for Text Classification. Retrieved June 11, 2022, from https://arxiv.org/abs/2107.03158

A Survey on Data Augmentation for Text Classification

Ding, Ning et al. “Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models.” ArXiv abs/2203.06904 (2022): n. pag. 
From  https://arxiv.org/pdf/2203.06904.pdf

Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models

Gou, J., Yu, B., Maybank, S.J., & Tao, D. (2021). Knowledge Distillation: A Survey. Int. J. Comput. Vis., 129, 1789-1819.

Link: https://arxiv.org/pdf/2006.05525.pdf

Learn Before

Related

Learn After