Learn Before
Ranking Stage in Top-k Sampling
The ranking stage is the second step in the Top-k sampling process, following expansion. In this stage, all potential next tokens generated by the language model are sorted in descending order based on their assigned probabilities. A predefined integer, 'k', determines how many of the top-ranked tokens are kept for the next stage. All other tokens with lower probabilities are pruned, or removed from consideration. For example, in the provided diagram where k=3, the tokens 'cute' (Pr=.34), 'on' (Pr=.32), and 'sick' (Pr=.21) are kept, while 'are' (Pr=.12) and '.' (Pr=.01) are pruned.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Expansion Stage in Top-k Sampling
Ranking and Pruning Stage in Top-k Sampling
A language model is generating the next word in a sentence and has calculated the probabilities for five potential words: 'house' (0.4), 'car' (0.3), 'boat' (0.15), 'plane' (0.1), and 'train' (0.05). The model uses a sampling method where it first ranks these words by probability, keeps only a specific number of the top-ranked words, renormalizes their probabilities to sum to 1, and then samples from this smaller set. How would decreasing the number of top-ranked words kept (e.g., from 4 to 2) most likely affect the generated text over time?
A language model is using a specific decoding method to generate the next token in a sequence. Arrange the following actions into the correct chronological order.
Ranking Stage in Top-k Sampling
Selection and Sampling Stage in Top-k Sampling
Output Stage in Top-k Sampling
Output Stage in Top-k Sampling
Applying a Probabilistic Filtering Method
Learn After
A language model has generated the following potential next tokens and their associated probabilities:
{'the': 0.45, 'a': 0.25, 'my': 0.15, 'your': 0.10, 'his': 0.05}. If this list of tokens proceeds to a filtering stage where only the top 3 most probable tokens are kept for further processing, which set of tokens will be selected?Diagnosing Repetitive Text Generation
Analyzing the Impact of 'k' on Token Pruning