Lee, Hung-Yi, et al. “Meta Learning for Natural Language Processing: A Survey.” ACL Anthology, https://aclanthology.org/2022.naacl-main.49/. 

Meta Learning for NLP

Meta-learning allows faster fine-tuning, converges to better performance, yields more generalizable models, and it achieves outstanding results for few-shot image classificaition. It alleviates the dependency of learning algorithms on labels and make model development more scalable.

Benefits of Meta-Learning

To learn meta-parameters φ, meta-training tasks Ttrain = {T1, T2, ..., TN } are required. where Tn is a task, and N is the number of tasks in Ttrain. Each task Tn includes a support set Sn and a query set Qn. Both Sn and Qn are paired examples as D in (1). The support set plays the role of training data in typical ML, while the query set can be understood as the testing data in typical ML.

Ttrain in Meta-Learining

In meta-learning, there is a loss function L(φ; Ttrain), which represents how “bad” a learning algorithm parameterized by φ is on Ttrain. L(φ; Ttrain) is the performance over all the tasks in Ttrain.

Loss Function in Meta-Learning

If φ is differentiable with respect to L(φ; Ttrain), then we can use gradient descent to learn meta-parameters; if not, we can use reinforcement learning algorithm or evolutionary algorithm. 

The Optimization Task in Meta Learning

Cross-domain transfer and cross-lingual transfer.

Two Types of Transfer in Meta Learning

The meta-training and meta-testing tasks can come from different problems. The intrinsic challenge in the cross-problem setting is that different NLP problems may need very different meta-parameters in learning algorithms. It's challenging to find unified meta-parameters on the meta-training tasks that can generalize to meta-testing tasks. And the meta-learning algorithms learning initial parameters require all the tasks to have a unified network architecture. 

Cross-problem Training

Meta-learning can also be used to realize domain generalization by learning an algorithm that can train from one domain but evaluate on the other. To simulate the domain generalization scenario, a set of meta-training tasks are constructed by sampling data from different domains as the support and query sets. With the meta-training tasks above, cross-task training will find the meta-parameters φ∗
that work well on the scenario where the training (support) and testing (query) examples are from different domains.

Domain Generalization

Meta-learning, also known as “learning to learn”,  aims to learn better learning algorithms, including better parameter initialization, optimization strategy, network architecture, distance metrics.

Hamilton College

Real-world relation distributions are long-tail: Only the common relations obtain sufficient training instances and most relations have very limited relational facts and corresponding sentences. Many relations even have less than 10 training instances. The general idea of few-shot models is to train good representations of instances or learn ways of fast adaptation from existing large-scale data, and then transfer to new tasks. 

Performing More Efficient Learning: Methods

Han, X. (n.d.). More Data, More Relations, More Context and More Openness:
A Review and Outlook for Relation Extraction *, Retrieved June 4, 2022, from https://arxiv.org/pdf/2004.03186.pdf

More Relations, More Context and More Openness:
A Review and Outlook for Relation Extraction

Metric learning learns a semantic metric on existing data and classifies queries by comparing them with training examples.

Learn Before

Related

Learn After