PaLM 2 is an updated version of the Pathway Language Model (PaLM) that scaled both the training data and model size at a roughly $$1:1$$ ratio. This scaling approach led to improved capabilities in multilingual tasks and reasoning.

PaLM 2

Minerva is a large language model created by further training a generalist model, specifically PaLM. It has demonstrated promising capabilities in solving quantitative and scientific reasoning problems.

Minerva

PaLM (Pathway Language Model) is a 540-billion-parameter Transformer decoder pretrained on 780 billion tokens. With its modified designs, it successfully outperformed average human performance on the BIG-Bench benchmark, continuing the trend of scaling language models.

Claude

Google

Decoder-only Transformers modify the original sequence-to-sequence Transformer architecture by completely removing the encoder component as well as the decoder sublayer responsible for encoder-decoder cross-attention. This streamlined architecture has become the de facto standard for large-scale language modeling, as it can effectively leverage vast amounts of unlabeled text corpora via self-supervised learning.

Decoder-Only Transformer Architecture

Dive into Deep Learning

The GPT (Generative Pre-Training) model represents an effort to design a general, task-agnostic architecture for context-sensitive representations. Built on a Transformer decoder, it pretrains a language model to represent text sequences. When adapted for downstream applications, the model's output feeds directly into an added linear layer to predict task labels. In sharp contrast to earlier models that freeze their pretrained weights, GPT fine-tunes all parameters in the pretrained Transformer decoder during supervised learning. Evaluated on twelve tasks encompassing natural language inference, question answering, sentence similarity, and classification, GPT improved the state of the art in nine of them with minimal changes to its core architecture.

Learn Before

Related

Learn After