Suppose there’s task $T_1,T_2,..,T_C$, each task has a training set $\{D_{train}^{c}, D_{testing}^{c}\}$, and some of them has larger training set, while some of them have smaller training set. The few shot tasks typically have less training set and are our target task. Multitask learning models are jointly trained on these tasks, i.e., parameters for one task are constrained by other tasks.

Multitask Learning

Embedding learning constrains the searching space $\mathcal{H}$ by mapping each input $x_i\in \mathbb{R}^n$ to a lower dimension space $z_i\in\mathbb{R}^m$, m<n. The mapping process is achieved by an embedding function $f(\cdot)$. And we can achieve better accuracy by using different embedding functions $f_{train}$ and $f_{test}$ for the training and testing set. The prediction on the testing set is made by finding the “closet” training sample compared to one testing sample in the embedded space and predicting the same label for this testing sample as the training sample.

Embedding Learning

Learning with external memory first maps the training sample to a lower dimension space using the an embedding function, and save the mapped value and class labels as key-value pair to computers. For a given testing sample, it’s first transformed to the same low dimension space, and then the most similar key-value pairs in computer memory are extracted and combined to form a representation of this testing sample. Lastly, a simple prediction function is used to predict the class of this testing sample.

Learning with External Memory

Generative modeling assumes that there exists a distribution $p(x)$ of x, and aims to estimate this distribution. This typically involve estimating $p(x|y), p(y)$. Generative modeling use prior knowledge by assuming that $p(x;\theta)$ is parameterized by $\theta$, and latent variable z with distribution $p(z;\gamma)$. So $p(x;\theta)$ can be estimated as integrating $p(x|z;\theta)p(z;\gamma)$ over $z$. $p(z;\gamma)$ is estimated using prior knowledge.

Generative Modeling

1)	Multitask Learning

2)	Embedding Learning

3)	Learning with External Memory

4)	Generative Modeling


University of Notre Dame

Few-Shot Learning can be divided by what part that prior knowledge has been added to:

1)	Data: The training data is augmented by prior knowledge

2)	Model: The possible model space is constrained according to prior knowledge

3)	Algorithm: The search strategy in hypothesis model space has been improved.

Another classification of Few-Shot Learning

Wang, Y., Yao, Q., Kwok, J. T., & Ni, L. M. (2020). Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3), 1-34.

Generalizing from a Few Examples: A Survey on Few-Shot Learning

Prior knowledge can be used to augment the training set, and below are several ways of achieving it:

1.	Transforming Samples from Training Set

2.	Transforming Samples from a Weakly Labeled or Unlabeled Data Set

3.	Transforming Samples from Similar Data Sets


Ways of Augmenting Data of Few-Shot Learning

Types of Few Shot Learning by Model

In terms of how searching strategy is affected by prior knowledge, searching algorithm motivated few-shot learning methods can be categorized as:

1)	Refining existing parameters

2)	Refining meta-learned parameters.

3)	Learning the optimizer


Learn Before

Related

Learn After