 In a CL setting, vocabulary may change with respect to two dimensions: time and domain. There is an established consensus that the meaning of words changes over time due to complicated linguistic and social processes. Hence, it is important to detect and accommodate shifts in meaning and data distribution, while preventing previously learned representations from CF.

CL in Word and Sentence Representations

In principle, a large LM trained on a sufficiently large and diverse corpus is able to perform well across many datasets and domains. Research has concluded that continuous domain- and task-adaptive pre-training of LMs leads to performance gains in downstream NLP tasks. As a result, further research interest in LM-based methods for CL in NLP has recently spiked.

CL in Language Modeling

In the context of continual learning, research in dialogue agents, which are able to continually learn, is a very active area. Research indicate that a conversational model initially trained with fixed data can improve itself, when it learns from interactions with humans in an on-line fashion.

CL in Question Answering

Arguably the most pressing problem of current approaches to SA is their poor performance on new domains. This issue is of utmost importance if one thinks about CL in sentiment classification. Research shows that CL can enable SA models to adapt to a large number of domains, since many new domains may already be covered by other past domains. Additionally, SA systems should become more accurate not only in classification but also in the discovery of word polarities in specific domains.

CL in Sentiment Analysis and Text Classification

The groundwork approach in this aspect is Neural Machine Translation (NMT), the corresponding research of which suggests that fine-tuning of the NMT model trained on out-of-domain data using a small in-domain parallel corpus boosts performance. However, NMT - as opposed to phrase-based MT - rarely incorporates translation memory, and so it is inherently harder for NMT models to adapt using active or interactive learning. Thus, CL for NMT is essential.

CL in Machine Translation

Continual learning has been applied extensively to the following NLP tasks:
- Word and sentence representations
- Language modeling
- Question answering
- Sentiment analysis and text classification
- Machine translation

University of Michigan - Ann Arbor

Continual learning (CL) is a machine learning paradigm, whose objective is to adaptively learn across time by leveraging previously learned tasks to improve generalization for future tasks. In other words, CL aims to enable information systems to learn from a continuous data stream across time.

Continual Learning

Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussà. 2020. Continual Lifelong Learning in Natural Language Processing: A Survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6523–6541.
https://aclanthology.org/2020.coling-main.574/

Continual Lifelong Learning in Natural Language Processing: A Survey

Catastrophic Forgetting (CF), or Catastrophic Inference, is the tendency of an artificial neural network to completely and abruptly forget previously learned information upon learning new information.

Catastrophic Forgetting (CF)

The majority of existing CL approaches tend to apply a single model structure to all tasks and control CF by various scheduling schemes. Three main family of methods are:
- Rehearsal methods
- Regularization methods
- Architectural methods

Learn Before

Related

Learn After