From the meta-training tasks Ttrain, cross-task training finds meta-parameters φ∗ parameterizing the learning algorithm Fφ∗ . With a sufficient number of tasks in Ttrain, cross-task training should find a suitable φ∗ for a wide range of domains, and thus also works well on the tasks in Ttest containing the domains
unseen during cross-task training. 

Cross-domain Transfer

If we consider different languages as different domains, then the cross-lingual transfer can be regarded as a special case of cross-domain transfer. A unified network architecture across all tasks is not obvious in cross-lingual learning because the vocabularies in different tasks are different. Before multilingual pre-trained models are available, unified word embeddings across languages are required.

 Cross-lingual Transfer

Cross-domain transfer and cross-lingual transfer.

Hamilton College

Meta-learning, also known as “learning to learn”,  aims to learn better learning algorithms, including better parameter initialization, optimization strategy, network architecture, distance metrics.

Meta Learning

Lee, Hung-Yi, et al. “Meta Learning for Natural Language Processing: A Survey.” ACL Anthology, https://aclanthology.org/2022.naacl-main.49/. 

Meta Learning for NLP

Meta-learning allows faster fine-tuning, converges to better performance, yields more generalizable models, and it achieves outstanding results for few-shot image classificaition. It alleviates the dependency of learning algorithms on labels and make model development more scalable.

Benefits of Meta-Learning

To learn meta-parameters φ, meta-training tasks Ttrain = {T1, T2, ..., TN } are required. where Tn is a task, and N is the number of tasks in Ttrain. Each task Tn includes a support set Sn and a query set Qn. Both Sn and Qn are paired examples as D in (1). The support set plays the role of training data in typical ML, while the query set can be understood as the testing data in typical ML.

Ttrain in Meta-Learining

In meta-learning, there is a loss function L(φ; Ttrain), which represents how “bad” a learning algorithm parameterized by φ is on Ttrain. L(φ; Ttrain) is the performance over all the tasks in Ttrain.

Loss Function in Meta-Learning

If φ is differentiable with respect to L(φ; Ttrain), then we can use gradient descent to learn meta-parameters; if not, we can use reinforcement learning algorithm or evolutionary algorithm. 

The Optimization Task in Meta Learning

Two Types of Transfer in Meta Learning

The meta-training and meta-testing tasks can come from different problems. The intrinsic challenge in the cross-problem setting is that different NLP problems may need very different meta-parameters in learning algorithms. It's challenging to find unified meta-parameters on the meta-training tasks that can generalize to meta-testing tasks. And the meta-learning algorithms learning initial parameters require all the tasks to have a unified network architecture. 

Cross-problem Training

Meta-learning can also be used to realize domain generalization by learning an algorithm that can train from one domain but evaluate on the other. To simulate the domain generalization scenario, a set of meta-training tasks are constructed by sampling data from different domains as the support and query sets. With the meta-training tasks above, cross-task training will find the meta-parameters φ∗
that work well on the scenario where the training (support) and testing (query) examples are from different domains.

Domain Generalization

In few-shot learning, meta-learning aims to learn effective parameter initializations and optimization strategies through the experience gained on meta-training data. This allows models to quickly adapt to new tasks using only a few examples.

Learn Before

Related

Learn After