Some examples of ablation studies: 
- No NSP: A bidirectional model which is still trained using the “Masked LM” but doesn’t include the “next sentence prediction” (NSP) task
- LTR and No NSP: Same as above but unidirectional, reads left to right and doesn’t use masking
- Trying to understand the effect of model size by building versions of BERT with varying numbers of layers, hidden units, etc but using the same hyperparameters and training procedure on each

University of Massachusetts at Amherst

Ablation studies on BERT are attempts to understand the relative importance of the various pieces of BERT’s functionality by testing them in isolation.

BERT Ablation Studies

Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

https://paperswithcode.com/method/bert

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

BERT Ablation Studies Examples

- A task-specific model architecture is added to improve the representation of the tasks.
- Running many experiments with cheaper models after pre-computing the expensive representation of the training data once is beneficial for the computation.


Learn Before

Related