State University of New York at Stony Brook

During the supervised learning phase for downstream tasks, the fine-tuning process for BERT shares two key similarities with GPT. First, the contextual representations generated by the pre-trained Transformer encoder are fed into an added output layer, requiring minimal modifications to the core architecture regardless of the task's nature (e.g., predicting a label for every token versus the entire sequence). Second, all parameters of the pre-trained model undergo fine-tuning, while the new parameters of the additional output layer are trained from scratch.

Similarities Between BERT and GPT in Fine-Tuning

Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

https://paperswithcode.com/method/bert

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Transfer Learning Method 1

Transfer Learning Method 2

A simple method for designing a self-supervised classification task to train an encoder is Next Sentence Prediction (NSP), as presented in the original BERT paper. This approach is built on the assumption that a good text encoder should effectively capture the relationship between two sentences. To model this, NSP uses the output of encoding two consecutive sentences, $$\mathrm{Sent}_{A}$$ and $$\mathrm{Sent}_{B}$$, to determine whether $$\mathrm{Sent}_{B}$$ is indeed the next sentence following $$\mathrm{Sent}_{A}$$. For example, if $$\mathrm{Sent}_{A}$$ is 'It is raining .' and $$\mathrm{Sent}_{B}$$ is 'I need an umbrella .', the model is tasked with recognizing this sequential relationship.

Learn Before

Related