Google

Pre-training paradigms differ primarily in their data requirements, underlying assumptions, and adaptation methods. Unsupervised pre-training utilizes large-scale unlabeled data to create a good preliminary starting point, though it still requires considerable effort to further train the model with labeled data. Supervised pre-training relies on labeled datasets from the outset, operating on the assumption that different tasks are related, which allows a model trained on one task to be transferred to another via tuning. In contrast, self-supervised pre-training leverages large-scale unlabeled data to train a model that can be efficiently adapted to new tasks through methods like fine-tuning or prompting.

Comparison of Pre-training Paradigms

A research team aims to develop a model that can answer questions about a highly specialized and new scientific domain. They have collected a massive corpus of research papers from this domain, but none of it is in a question-and-answer format. The team has the resources to manually create a small, high-quality dataset of 1,000 question-answer pairs. Given the available data and the team's goal, which combination of initial training and subsequent adaptation methods would be most effective and resource-efficient? Justify your choice.

Selecting a Model Training Strategy

Match each pre-training paradigm with the description that best characterizes its data requirements and common adaptation methods.

A research lab has access to a vast corpus of unlabeled text from the internet but has a very limited budget for creating task-specific labeled datasets. Their goal is to develop a foundational model that can be flexibly adapted to a wide variety of future tasks, often with only a few examples for each new task. Which pre-training paradigm would be the most strategic choice for this lab?

Learn Before

Related