1)	Refining Representations: $D_{train}$ is put into memory such that the key-value pairs can represent $x_{test}$ better.

2)	Refining Parameters: the parameters of the embedding function of the testing set are refined.


Types of Learning with External Memory by its Functionality

Learning with external memory first maps the training sample to a lower dimension space using the an embedding function, and save the mapped value and class labels as key-value pair to computers. For a given testing sample, it’s first transformed to the same low dimension space, and then the most similar key-value pairs in computer memory are extracted and combined to form a representation of this testing sample. Lastly, a simple prediction function is used to predict the class of this testing sample.

University of Notre Dame

Google

1)	Multitask Learning

2)	Embedding Learning

3)	Learning with External Memory

4)	Generative Modeling


Types of Few Shot Learning by Model

Wang, Y., Yao, Q., Kwok, J. T., & Ni, L. M. (2020). Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3), 1-34.

Generalizing from a Few Examples: A Survey on Few-Shot Learning

Reference of Foundations of Large Language Models Course

Suppose there’s task $T_1,T_2,..,T_C$, each task has a training set $\{D_{train}^{c}, D_{testing}^{c}\}$, and some of them has larger training set, while some of them have smaller training set. The few shot tasks typically have less training set and are our target task. Multitask learning models are jointly trained on these tasks, i.e., parameters for one task are constrained by other tasks.

Multitask Learning

Embedding learning constrains the searching space $\mathcal{H}$ by mapping each input $x_i\in \mathbb{R}^n$ to a lower dimension space $z_i\in\mathbb{R}^m$, m<n. The mapping process is achieved by an embedding function $f(\cdot)$. And we can achieve better accuracy by using different embedding functions $f_{train}$ and $f_{test}$ for the training and testing set. The prediction on the testing set is made by finding the “closet” training sample compared to one testing sample in the embedded space and predicting the same label for this testing sample as the training sample.

Embedding Learning

Learning with External Memory

Generative modeling assumes that there exists a distribution $p(x)$ of x, and aims to estimate this distribution. This typically involve estimating $p(x|y), p(y)$. Generative modeling use prior knowledge by assuming that $p(x;\theta)$ is parameterized by $\theta$, and latent variable z with distribution $p(z;\gamma)$. So $p(x;\theta)$ can be estimated as integrating $p(x|z;\theta)p(z;\gamma)$ over $z$. $p(z;\gamma)$ is estimated using prior knowledge.

Learn Before

Related

Learn After