1)	Task-Specific Embedding Model: embedding functions are different for different tasks and training for a task-specific embedding function only use training data from that specific task.

2)	Task-Invariant Embedding Model: embedding functions are the same for different tasks, and embedding function training is on a large-scale dataset with enough samples.

3)	Hybrid Embedding Model: embedding functions learned from the generic model is adapted by task-specific data.


Types of Embedding Model

Embedding learning constrains the searching space $\mathcal{H}$ by mapping each input $x_i\in \mathbb{R}^n$ to a lower dimension space $z_i\in\mathbb{R}^m$, m<n. The mapping process is achieved by an embedding function $f(\cdot)$. And we can achieve better accuracy by using different embedding functions $f_{train}$ and $f_{test}$ for the training and testing set. The prediction on the testing set is made by finding the “closet” training sample compared to one testing sample in the embedded space and predicting the same label for this testing sample as the training sample.

University of Notre Dame

1)	Multitask Learning

2)	Embedding Learning

3)	Learning with External Memory

4)	Generative Modeling


Types of Few Shot Learning by Model

Wang, Y., Yao, Q., Kwok, J. T., & Ni, L. M. (2020). Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3), 1-34.

Generalizing from a Few Examples: A Survey on Few-Shot Learning

Suppose there’s task $T_1,T_2,..,T_C$, each task has a training set $\{D_{train}^{c}, D_{testing}^{c}\}$, and some of them has larger training set, while some of them have smaller training set. The few shot tasks typically have less training set and are our target task. Multitask learning models are jointly trained on these tasks, i.e., parameters for one task are constrained by other tasks.

Multitask Learning

Embedding Learning

Learning with external memory first maps the training sample to a lower dimension space using the an embedding function, and save the mapped value and class labels as key-value pair to computers. For a given testing sample, it’s first transformed to the same low dimension space, and then the most similar key-value pairs in computer memory are extracted and combined to form a representation of this testing sample. Lastly, a simple prediction function is used to predict the class of this testing sample.

Learning with External Memory

Generative modeling assumes that there exists a distribution $p(x)$ of x, and aims to estimate this distribution. This typically involve estimating $p(x|y), p(y)$. Generative modeling use prior knowledge by assuming that $p(x;\theta)$ is parameterized by $\theta$, and latent variable z with distribution $p(z;\gamma)$. So $p(x;\theta)$ can be estimated as integrating $p(x|z;\theta)p(z;\gamma)$ over $z$. $p(z;\gamma)$ is estimated using prior knowledge.

Learn Before

Related

Learn After