Google

GPT-1, introduced by Radford et al. in 2018, is a generative pre-trained transformer model. A key contribution of this model was the introduction of the multi-head self-attention mechanism and the use of a transformer architecture instead of RNNs or CNNs, which set a new state-of-the-art. The model's loss function is based on approximating the probability of sentence construction.

GPT-1 (Generative Pre-trained Transformer)

A foundational generative language model introduced in 2018 significantly improved the ability to capture relationships between words far apart in a text, a major challenge for previous sequential models. Which of the following best analyzes the core architectural innovation responsible for this leap in performance?

A pioneering 2018 language model was based on a transformer architecture that processed text strictly in a left-to-right sequence to predict the next word. Evaluate the primary conceptual limitation of this unidirectional approach. In your analysis, discuss specific types of language understanding tasks where this design choice would likely result in suboptimal performance and explain why.

Critique of an Early Transformer-Based Language Model

A pioneering generative language model from 2018, which was the first to be based on the transformer architecture, was trained using a specific objective on a large corpus of unlabeled text. In your own words, describe the primary goal of this model's pre-training process.

Learn Before

Related