Memory networks were first introduced because neural networks lacked a working memory. They have a set of memory cells that can be easily accessed via an addressing mechanism.

Memory Network

Although the terms 'short-term memory' and 'working memory' are frequently used interchangeably, they represent distinct psychological concepts. Short-term memory specifically refers to the temporary storage system holding incoming sensory information. In contrast, working memory is a more comprehensive system that encompasses this temporary storage. Consequently, short-term memory is most accurately conceptualized as a sub-component operating within the larger working memory framework.

Distinction Between Short-Term Memory and Working Memory

Short-term memory (STM) serves as a temporary storage system that processes incoming information from sensory memory. It is capable of retaining a limited amount of data for a brief duration of 15 to 30 seconds. While in this active state, short-term memory can also integrate new sensory input with existing knowledge retrieved from long-term memory. Although often confused with working memory, short-term memory is more accurately classified as a distinct sub-component within the broader working memory framework.

Short-term Memory (STM)

The working memory is the system that enables the ability to explicitly hold and manipulate pieces of information that are used to achieve a goal. Unlike humans, neural networks do not have a working memory.

University of Michigan - Ann Arbor

Ohio State University, Columbus

Google

https://www.coursera.org/learn/neural-networks-deep-learning?specialization=deep-learning

Neural Networks and Deep Learning

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Spielman, R. M., Jenkins, W. J., & Lovett, M. D. (2020, April 22). Psychology (2nd ed.). OpenStax. https://openstax.org/books/psychology-2e/pages/1-introduction

OpenStax Psychology (2nd ed.) Textbook

(1) Standard neural networks: real estate, online advertising

(2) Convolutional neural networks: applied in image recognition including photo tagging

(3) Recurrent Neural Networks: applied in sequence data including speech recognition, machine translation

(4) custom/hybrid neural networks: complex applications such as autonomous driving

application of neural networks in supervised learning

http://datahacker.rs/neural-networks-representation/

deep-learning representation

Because we have access to large columns of data, more computational power, and advanced algorithms.

For big data, large neural networks perform better than traditional ML models (SVM, logistic regression...)

Why is deep learning taking off?

(1) initialize parameters
(2) Implement the forward propagation to get the prediction
(3) compute the loss
(4) Implement the backward propagation to compute the derivatives of the parameters
(5) update the parameters
(6) repeat steps (2) to (5) to reduce the cost.

The learning circle of the neural network

- Start off trying to have one neural network do multiple things (can actually lead to better performance)
- differences and similarities amongst tasks is utilized to output a better result

Multi-task Learning

Benchmarking is a way to analyze how well your neural network is performing on a given standard set.
-Ensembling
-Multicrop at test time
-using open source code


Benchmarking

• Single neuron == linear regression without applying activation(perceptron)

• Basically a single neuron will calculate weighted sum of input(W.T*X) and then we can set a threshold to predict output in a perceptron. If weighted sum of input cross the threshold, perceptron fires and if not then perceptron doesn’t predict.

What is a Neural Network(NN)?

Working Memory

Beyond standard categorical and numeric variables, deep learning practitioners frequently work with myriad unstructured data formats. These complex data types include text strings for natural language processing, images and point clouds for computer vision, and audio data. Handling these diverse formats requires specialized approaches and advanced tools compared to traditional tabular data.

Learn Before

Related

Learn After