Rehearsal methods rely on retaining some training examples from prior tasks, so that they can later be shown to a task at hand. As training samples are kept per each task and are periodically replayed while learning the model, the computing and memory requirements of the model increase proportionally to the number of tasks.

Rehearsal Methods

Regularization methods are single-model approaches that rely on a fixed model capacity with an additional loss term that aids knowledge consolidation while learning subsequent tasks or data distributions.

Regularization Methods

Architectural methods prevent forgetting by applying modular changes to the network’s architecture and introducing task-specific parameters. Typically, previous task parameters are kept fixed or masked out. Moreover, new layers are often injected dynamically to augment a model with additional modules to accommodate new tasks.

Architectural Methods

The majority of existing CL approaches tend to apply a single model structure to all tasks and control CF by various scheduling schemes. Three main family of methods are:
- Rehearsal methods
- Regularization methods
- Architectural methods

University of Michigan - Ann Arbor

Continual learning (CL) is a machine learning paradigm, whose objective is to adaptively learn across time by leveraging previously learned tasks to improve generalization for future tasks. In other words, CL aims to enable information systems to learn from a continuous data stream across time.

Continual Learning

Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussà. 2020. Continual Lifelong Learning in Natural Language Processing: A Survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6523–6541.
https://aclanthology.org/2020.coling-main.574/

Continual Lifelong Learning in Natural Language Processing: A Survey

Catastrophic Forgetting (CF), or Catastrophic Inference, is the tendency of an artificial neural network to completely and abruptly forget previously learned information upon learning new information.

Catastrophic Forgetting (CF)

Continual Learning Approaches

Continual learning has been applied extensively to the following NLP tasks:
- Word and sentence representations
- Language modeling
- Question answering
- Sentiment analysis and text classification
- Machine translation

Learn Before

Related

Learn After