Architectural methods prevent forgetting by applying modular changes to the network’s architecture and introducing task-specific parameters. Typically, previous task parameters are kept fixed or masked out. Moreover, new layers are often injected dynamically to augment a model with additional modules to accommodate new tasks.

University of Michigan - Ann Arbor

The majority of existing CL approaches tend to apply a single model structure to all tasks and control CF by various scheduling schemes. Three main family of methods are:
- Rehearsal methods
- Regularization methods
- Architectural methods

Continual Learning Approaches

Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussà. 2020. Continual Lifelong Learning in Natural Language Processing: A Survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6523–6541.
https://aclanthology.org/2020.coling-main.574/

Continual Lifelong Learning in Natural Language Processing: A Survey

Rehearsal methods rely on retaining some training examples from prior tasks, so that they can later be shown to a task at hand. As training samples are kept per each task and are periodically replayed while learning the model, the computing and memory requirements of the model increase proportionally to the number of tasks.

Rehearsal Methods

Regularization methods are single-model approaches that rely on a fixed model capacity with an additional loss term that aids knowledge consolidation while learning subsequent tasks or data distributions.

Learn Before

Related