Analyzing Repetitive Model Output
A language model is generating a story and produces the output shown below. Based on the principle of calculating the probability of the next word given the preceding context, analyze this output and explain the most likely probabilistic reason why the model has entered this repetitive loop.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A text generation model is tasked with completing the sentence: 'The mountain climber reached the summit and felt a sense of'. The model calculates the probability of several potential next words based on the preceding text. Given the following calculated probabilities, which word will the model select to continue the sequence?
Analyzing Repetitive Model Output
A language model generates text by calculating the probability of the next word given all the preceding words. Consider the following two contexts:
Context A: 'The chef carefully seasoned the soup. He tasted it and decided it needed more' Context B: 'The comedian carefully timed the joke. He tested it and decided it needed more'
Which statement best analyzes the likely probability the model would assign to the word 'salt' as the very next word?