Learn Before
Mathematical Representation of the Top-p Candidate Pool
In top-p (nucleus) sampling, the candidate pool at a given step , denoted as , is composed of the most probable tokens. The value of is the size of the smallest set of top-ranked tokens whose cumulative probability meets or exceeds the threshold . The pool is formally represented as the set of these top tokens:
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Mathematical Representation of the Top-p Candidate Pool
A language model is generating the next word and has calculated the following probabilities for the most likely tokens: Token A (0.40), Token B (0.30), Token C (0.15), Token D (0.10), and Token E (0.05). If the model uses a sampling strategy where it forms a candidate pool by including the most probable tokens until their cumulative probability just exceeds a threshold of 0.75, what will be the size of this candidate pool?
Relationship Between Threshold and Candidate Pool Size
A language model is generating the next token in two different contexts. In both contexts, the model uses a sampling method where it forms a candidate pool by selecting the smallest set of the most probable tokens whose cumulative probability exceeds a threshold of 0.9.
- Context A: The single most probable token has a probability of 0.95.
- Context B: The ten most probable tokens each have a probability of 0.09.
How will the size of the candidate token pool compare between these two contexts?
Learn After
At a specific step 'i' in a text generation process, the model has calculated the following probabilities for the next token from a vocabulary of {A, B, C, D, E}:
P(A) = 0.40 P(B) = 0.30 P(C) = 0.15 P(D) = 0.10 P(E) = 0.05
If the sampling process uses a probability threshold 'p' of 0.8, which of the following sets correctly represents the candidate pool of tokens, denoted as ?
Constructing the Top-p Candidate Pool
A language model's output probabilities for the next token are sorted in descending order. The candidate pool for sampling, represented as , is constructed by including all tokens whose individual probability is greater than the sampling threshold .