BERT is applied to the CoNLL-2003 Named Entity Recognition (NER) task. From the table we can see that there is only 0.3 F1 difference between the best performing method that concatenates the token representations from the top four hidden layers of the pre-trained Transformer and the one fine-tuningthe entire model. Therefore, BERT is effective for both fine-tuning and feature-based approaches.

Comparison Between Two Approaches

- A task-specific model architecture is added to improve the representation of the tasks.
- Running many experiments with cheaper models after pre-computing the expensive representation of the training data once is beneficial for the computation.


University of Michigan - Ann Arbor

Ohio State University, Columbus

Ablation studies on BERT are attempts to understand the relative importance of the various pieces of BERT’s functionality by testing them in isolation.

BERT Ablation Studies

Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

https://paperswithcode.com/method/bert

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Some examples of ablation studies: 
- No NSP: A bidirectional model which is still trained using the “Masked LM” but doesn’t include the “next sentence prediction” (NSP) task
- LTR and No NSP: Same as above but unidirectional, reads left to right and doesn’t use masking
- Trying to understand the effect of model size by building versions of BERT with varying numbers of layers, hidden units, etc but using the same hyperparameters and training procedure on each

Learn Before

Related

Learn After