Although gathering as much training data as possible is generally desirable for Large Language Models, simply increasing the size of the dataset does not guarantee better training results. This dynamic introduces new challenges in creating and collecting effective datasets.

Google

As neural networks are scaled up, their demand for data increases significantly. Developing Large Language Models requires pre-training on massive datasets, often containing trillions of tokens, which is orders of magnitude larger than the data used to train conventional Natural Language Processing models.

Data Demand for Large Language Models

Reference of Foundations of Large Language Models Course

Data Volume vs. Quality in LLM Pre-training

GPT-3 is a generative pre-trained transformer model introduced in 2020. Its largest version, GPT3-175B, contains 175 billion parameters and required 0.5 trillion tokens of pre-training data sourced from webpages, books, and Wikipedia.

GPT-3

Falcon is a family of large language models. A notable version is Falcon-180B, which contains 180 billion parameters and was pre-trained on 3.5 trillion tokens derived from diverse sources including webpages, books, conversations, code, and technical articles.

Falcon

LLaMA2 is a family of large language models introduced in 2023. It includes prominent versions such as the 65-billion-parameter LLaMA2-65B, which was pre-trained on between 1.0 trillion and 1.4 trillion tokens. The training data for LLaMA2 comes from a diverse mix of public sources, including webpages, software code, Wikipedia, books, academic papers, and question-and-answer content.

LLaMA2

PaLM-450B is a large language model introduced in 2022. It was pre-trained on a massive corpus of 0.78 trillion tokens, utilizing diverse data sources that include webpages, books, conversations, code, Wikipedia, and news articles.

PaLM-450B

Gemma-7B is a large language model pre-trained on a massive dataset of 6 trillion tokens. Its pre-training data is sourced from webpages, mathematics content, and code.

Learn Before

Related