Learn Before
Relationship Between Threshold and Candidate Pool Size
A language model developer is adjusting the threshold 'p' in a text generation system that uses a sampling method based on cumulative probability. Explain the relationship between increasing the value of 'p' (e.g., from 0.7 to 0.9) and the size of the resulting candidate pool of tokens (kp). Will kp always increase, always decrease, or is the relationship more nuanced? Justify your reasoning.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Mathematical Representation of the Top-p Candidate Pool
A language model is generating the next word and has calculated the following probabilities for the most likely tokens: Token A (0.40), Token B (0.30), Token C (0.15), Token D (0.10), and Token E (0.05). If the model uses a sampling strategy where it forms a candidate pool by including the most probable tokens until their cumulative probability just exceeds a threshold of 0.75, what will be the size of this candidate pool?
Relationship Between Threshold and Candidate Pool Size
A language model is generating the next token in two different contexts. In both contexts, the model uses a sampling method where it forms a candidate pool by selecting the smallest set of the most probable tokens whose cumulative probability exceeds a threshold of 0.9.
- Context A: The single most probable token has a probability of 0.95.
- Context B: The ten most probable tokens each have a probability of 0.09.
How will the size of the candidate token pool compare between these two contexts?