Learn Before
Impact of Selection Pool Size on Text Generation
Two language models, Model A and Model B, are at the same point in generating a story and have produced the exact same probability distribution for the next token. Model A is configured to choose from only the 2 most likely candidates, while Model B is configured to choose from the 5 most likely candidates. Based on this difference, which model is more likely to generate a surprising or less conventional continuation, and why does the size of its candidate pool lead to this outcome?
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Formal Derivation of the Top-k Selection Pool
A language model is generating text and has calculated the following probabilities for the next potential token:
{'the': 0.45, 'a': 0.20, 'cat': 0.12, 'dog': 0.08, 'ran': 0.07, 'jumped': 0.05}. If the model is configured to sample its next choice from only the 4 most likely candidates, which set of tokens constitutes the selection pool?Impact of Selection Pool Size on Text Generation
When generating text by sampling from a pool of the most probable candidate tokens, setting the pool size to 1 will produce the exact same output sequence as a method that always deterministically chooses the single token with the highest probability at every step.