State University of New York at Stony Brook

BERT&GPT and Fine Tuning

Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

https://paperswithcode.com/method/bert

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Transfer Learning Method 1

Transfer Learning Method 2

A simple method for designing a self-supervised classification task to train an encoder is Next Sentence Prediction (NSP), as presented in the original BERT paper. This approach is built on the assumption that a good text encoder should effectively capture the relationship between two sentences. To model this, NSP uses the output of encoding two consecutive sentences, $$\mathrm{Sent}_{A}$$ and $$\mathrm{Sent}_{B}$$, to determine whether $$\mathrm{Sent}_{B}$$ is indeed the next sentence following $$\mathrm{Sent}_{A}$$. For example, if $$\mathrm{Sent}_{A}$$ is 'It is raining .' and $$\mathrm{Sent}_{B}$$ is 'I need an umbrella .', the model is tasked with recognizing this sequential relationship.

Learn Before

Related