Training Decoder-Only Language Models with Cross-Entropy Loss

Output Probability Calculation in Transformer Language Models

Global Nature of Standard Transformer LLMs

Processing Flow of Autoregressive Generation in a Decoder-Only Transformer

Initial Input Representation for Transformer Layers

Greedy Decoding in Language Models

Structure of a Transformer Block

A generative language model is designed to produce text by predicting the next token based solely on the sequence of tokens that came before it. If you were to adapt a standard Transformer decoder block for this specific auto-regressive task, which of its sub-layers would you remove, and why is this modification functionally necessary?

A language model is constructed using a stack of modified Transformer decoder blocks. Each block contains a self-attention sub-layer and a feed-forward network sub-layer, but lacks the sub-layer that would process information from a separate, secondary input sequence. This model is capable of performing a machine translation task, such as translating a German sentence into English, without any further architectural changes.

Function of Self-Attention in Auto-regressive Generation

Neural Network-Based Next-Token Probability Distribution

Self-Attention layer understanding - Step 5 - Adding the time

Input Embedding with Positional Encoding

Learnable Absolute Positional Embeddings

Initial Input Representation for Transformer Layers

Comparison of Arbitrary Order Prediction and Masked Language Modeling

An engineer builds a language model where all input words in a sentence are processed simultaneously and independently before their information is combined. When testing the model with the sentences 'The cat chased the dog' and 'The dog chased the cat', the engineer observes that the model generates identical internal representations for both, failing to capture their different meanings. Which of the following modifications would most directly address this fundamental flaw?

Model Architecture Design Choice

Analyzing Order-Insensitivity in Language Models