This example illustrates the K-best selection process with a beam width (K) of 3. Given five candidate words with their respective probabilities—'cute' (Pr=0.34), 'on' (Pr=0.32), 'sick' (Pr=0.21), 'are' (Pr=0.12), and '.' (Pr=0.01)—the process involves two steps. First, the candidates are ranked by their probability scores. Second, the top K=3 candidates ('cute', 'on', 'sick') are selected as the output, forming the beam. The remaining candidates with lower scores ('are', '.') are discarded or 'pruned'.

Example of K-Best Selection with a Beam Width of 3

In beam search, after the expansion stage generates potential next tokens, the ranking stage sorts these candidates based on their assigned probabilities. For example, as shown in the diagram, the tokens 'cute', 'on', 'sick', 'are', and '.' are ranked according to their probabilities of 0.34, 0.32, 0.21, 0.12, and 0.01, respectively.

Ranking Stage in Beam Search

The expansion stage is the initial step in the K-best selection process. In this phase, a language model generates a set of potential next tokens or words based on the current context. For example, as depicted in the diagram labeled '① Expansion', given the token 'is', the model might produce a list of candidate words such as '.', 'cute', 'on', 'are', and 'sick'. These candidates are then passed to the subsequent ranking stage.

Expansion Stage in K-Best Selection

A language model generates the following potential next words and their corresponding probabilities: 'house' (0.25), 'car' (0.40), 'boat' (0.15), 'plane' (0.18), and 'train' (0.02). If a selection process is used to keep only the top 3 most probable words, which set of words will be chosen?

A language model is generating the next word in a sentence. Arrange the following actions into the correct sequence for selecting the most promising candidates.

The ranking stage, labeled as step ② in the process, follows the expansion of candidate tokens. In this step, all generated candidates are sorted in descending order based on their probabilities. For example, the candidates 'cute' (Pr=0.34), 'on' (Pr=0.32), 'sick' (Pr=0.21), 'are' (Pr=0.12), and '.' (Pr=0.01) are ranked. This stage prepares for the final selection by identifying which candidates will be kept and which will be discarded or 'pruned'.

Ranking Stage in K-Best Selection

The output stage, labeled as step ③, is the final part of the K-best selection process. Based on a predefined 'beam width' (K), the top K candidates from the ranked list are selected as the final output. All other candidates with lower probabilities are discarded. For instance, with a beam width of K=3, the top three candidates ('cute', 'on', 'sick') are chosen, forming the final output beam.

Output Stage in K-Best Selection

A developer is tuning a language model for creative story writing. They are testing two configurations for the word selection process:

*   **Configuration X:** At each step, the model keeps only the single most probable next word.
*   **Configuration Y:** At each step, the model keeps the top 10 most probable next words.

Evaluate which configuration is more likely to produce a creative and less repetitive story. Justify your answer by explaining the trade-offs involved in the selection process.

Impact of Selection Parameter on Text Generation

Google

The flowchart helps work out different scenarios or use cases that the design of a system or service must take into account, especially when the process can go one way or another at an important decision points.

4) Flowcharts

Reference of Foundations of Large Language Models Course

The Ranking and 1-Best Selection process is a decoding method that involves three main stages. First, in the Expansion stage, a set of potential next tokens is generated. Second, in the Ranking stage, these candidates are sorted based on their probabilities from highest to lowest. Finally, in the Output stage, the single token with the highest probability (the '1-best') is selected, and all other lower-probability tokens are discarded or 'pruned'.

Ranking and 1-Best Selection Process

Ranking and K-Best Selection Process

Top-k sampling is a text generation strategy that balances creativity and coherence. The process involves several stages: first, an 'Expansion' stage generates all possible next tokens; second, a 'Ranking' stage filters this list to only the top 'k' most probable tokens; third, a 'Selection and Sampling' stage renormalizes the probabilities of these k tokens and samples one; and finally, an 'Output' stage finalizes the sampled token as the next word in the sequence.

Top-K Sampling Process

Top-p, or nucleus, sampling is a probabilistic text generation technique that involves a multi-stage process. First, in the expansion stage, all potential next tokens are generated. Second, these tokens are ranked by probability. Third, a 'nucleus' of the top-ranked tokens is selected, such that their cumulative probability exceeds a predefined threshold 'p'. The probabilities within this nucleus are then renormalized. Finally, a single token is sampled from this renormalized set to become the output. This method balances quality and diversity by filtering out the long tail of low-probability tokens.

Learn Before

Related

Learn After