San Jose State University

The process of generating text with a language model involves two main stages. First, an initial sequence of tokens, referred to as the context or prefix and denoted by $x$, is established. Following this, the model proceeds to generate a subsequent sequence of tokens, known as the continuation and denoted by $y$, based on the provided initial input.

Text Generation from an Initial Context

Automatic Detection of Machine Generated Text: A Critical Survey
by Ganesh Jawahar, Muhammad Abdul-Mageed, Laks V.S. Lakshmanan
https://arxiv.org/pdf/2011.01314.pdf

Automatic Detection of Machine Generated Text: A Critical Survey


 - News article generation task 
 - Story generation task


Examples of text generation

Decoding Methods to Generate Continuations in TGM

Selecting random token based on the probability of previous token. There are two types 
 - top-k sampling
 - top-p (or nucleus) sampling 


Stochastic decoding methods in TGM

In the initial stage of text generation, all tokens that constitute the input context, denoted as 'x', are provided to the model simultaneously for parallel processing.

Simultaneous Processing of Input Context Tokens

A foundational step in sequence modeling is to process the input sequence to create an encoded representation, often referred to as the context. This representation synthesizes the information contained within the input tokens and serves as the basis for subsequent processing, such as generating an output.

Building the Encoded Representation of Input

A user gives a language model the input: "Ancient Rome was a civilization known for its". The model then produces the following output: "engineering marvels, such as aqueducts and roads." Based on the two-stage process of text generation, which statement best analyzes this interaction?

Arrange the following stages into the correct sequence that describes how a language model generates text based on an initial input.

In the context of the two-stage text generation process, identify the 'initial context' (or prefix) and the 'continuation' in the scenario below. Explain the role each part plays in how the model produces the final output.

Learn Before

Related