Catastrophic Forgetting (CF), or Catastrophic Inference, is the tendency of an artificial neural network to completely and abruptly forget previously learned information upon learning new information.

University of Michigan - Ann Arbor

Continual learning (CL) is a machine learning paradigm, whose objective is to adaptively learn across time by leveraging previously learned tasks to improve generalization for future tasks. In other words, CL aims to enable information systems to learn from a continuous data stream across time.

Continual Learning

Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussà. 2020. Continual Lifelong Learning in Natural Language Processing: A Survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6523–6541.
https://aclanthology.org/2020.coling-main.574/

Continual Lifelong Learning in Natural Language Processing: A Survey

Catastrophic Forgetting (CF)

The majority of existing CL approaches tend to apply a single model structure to all tasks and control CF by various scheduling schemes. Three main family of methods are:
- Rehearsal methods
- Regularization methods
- Architectural methods

Continual Learning Approaches

Continual learning has been applied extensively to the following NLP tasks:
- Word and sentence representations
- Language modeling
- Question answering
- Sentiment analysis and text classification
- Machine translation

Learn Before

Related