Google

Selecting random token based on the probability of previous token. There are two types 
 - top-k sampling
 - top-p (or nucleus) sampling 


Stochastic decoding methods in TGM

Top-k sampling is a decoding strategy where, at each step of the text generation process, the next token is selected by sampling from a reduced set of candidates. This set is limited to the 'k' tokens that have the highest predicted probabilities.

Top-k Sampling

Top-p sampling, also known as nucleus sampling, is a decoding method that selects the next token from a dynamically sized candidate pool. This pool is formed by identifying the smallest set of the most probable tokens whose cumulative probability exceeds a predefined threshold 'p' [Holtzman et al., 2020]. By constructing the candidate pool in this manner, the method avoids selecting low-probability tokens from the long tail of the distribution, which helps prevent the generation of incoherent or nonsensical text.

Top-p (Nucleus) Sampling

A team developing a language model for creative storytelling finds that its generated text is often repetitive and predictable, frequently getting stuck in loops (e.g., 'I am I am I am...'). Which of the following decoding strategies would be most effective at addressing this issue by introducing more variety into the generated text?

A language model was given the same starting phrase twice and produced two different continuations, labeled A and B. One was generated by always picking the single most probable next word, while the other was generated by randomly selecting from a set of high-probability words. Analyze the two outputs and determine which one (A or B) was likely generated using the random selection method. Justify your reasoning by pointing to specific characteristics of the text.

Analyzing Text Generation Outputs

In the context of a model generating text one token at a time, contrast a method that *always* selects the single most probable next token with a method that *randomly* selects from a distribution of likely next tokens. What is a primary advantage of the random selection approach for generating human-like text?

Comparing Text Generation Strategies

When using a stochastic decoding method for text generation, the model is guaranteed to select the single token with the highest probability at each step.

Learn Before

Related