Google

A fundamental distinction between pre-training and fine-tuning lies in the scale of data required. While more fine-tuning data is generally beneficial, the amount needed is orders of magnitude smaller than what is required for pre-training. For instance, fine-tuning can be effectively performed with tens or hundreds of thousands of samples, or even fewer if the data is of high quality. In contrast, pre-training models demands billions or even trillions of tokens, which consequently results in significantly larger computational requirements and longer training times.

Data Scale Disparity: Pre-training vs. Fine-tuning

Evaluate the CTO's proposal. Is this the most effective strategy for the startup? Justify your reasoning based on the typical data requirements for creating and adapting large models.

Evaluating a Data Strategy for a Specialized AI Model

A small startup is building a specialized legal document analysis tool. They plan to adapt a large, general-purpose pre-trained language model for this task. Given their limited budget and resources, which of the following data strategies is most likely to lead to a successful and efficient outcome?

A machine learning team is adapting a large, pre-trained language model for a specialized medical text summarization task. They have two potential datasets for this adaptation process:

*   **Dataset A:** 5 million general medical articles of mixed quality and relevance.
*   **Dataset B:** 20,000 high-quality summaries of medical articles, written and verified by expert physicians.

Analyze the two datasets and justify which one is the more strategic choice for the team's goal. In your justification, explain the relationship between data quantity, data quality, and this specific phase of model development.

Learn Before

Related