PaLM 2 is an updated version of the Pathway Language Model (PaLM) that scaled both the training data and model size at a roughly $$1:1$$ ratio. This scaling approach led to improved capabilities in multilingual tasks and reasoning.

PaLM 2

Minerva is a large language model created by further training a generalist model, specifically PaLM. It has demonstrated promising capabilities in solving quantitative and scientific reasoning problems.

Minerva

PaLM (Pathway Language Model) is a $$540$$-billion-parameter Transformer decoder pretrained on $$780$$ billion tokens. With its modified designs, it successfully outperformed average human performance on the BIG-Bench benchmark, continuing the trend of scaling language models.

Claude

Decoder-only Transformers modify the original sequence-to-sequence Transformer architecture by completely removing the encoder component as well as the decoder sublayer responsible for encoder-decoder cross-attention. This streamlined architecture has become the de facto standard for large-scale language modeling, as it can effectively leverage vast amounts of unlabeled text corpora via self-supervised learning.

Decoder-Only Transformer Architecture

Dive into Deep Learning

The GPT (Generative Pre-Training) model utilizes a Transformer decoder as its foundational architecture. It is trained via an autoregressive language modeling objective and consists of $$100$$ million parameters. Unlike subsequent models, GPT typically requires task-specific fine-tuning to perform effectively on individual downstream tasks.

Learn Before

Related

Learn After