Learn Before
Analyzing Component Influence in a k-NN Language Model
A k-NN augmented language model is predicting the next word. The base model strongly predicts 'mat', while the retrieved examples strongly suggest 'roof'. Given the probabilities below and an interpolation coefficient of 0.8, which word will the combined model most likely predict? Justify your answer by calculating the final probability for both 'mat' and 'roof' and explaining which model component had a greater influence on the outcome.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Next Token Selection in k-NN Language Models
A language model's final probability for a word is determined by blending its own internal prediction with a prediction based on retrieved text examples. The formula used is:
Final_Prob = 位 * Retrieved_Prob + (1 - 位) * Internal_Prob. In a scenario where the model's internal prediction for the next word is 'innovative', but the most frequent word in similar retrieved examples is 'creative', how would the value of the coefficient位influence the outcome?Analyzing Component Influence in a k-NN Language Model
In a language model that combines its own predictions with information from retrieved examples using the formula
Final_Prob = 位 * Retrieved_Prob + (1 - 位) * Base_LM_Prob, setting the coefficient位to 0 results in the final prediction being determined entirely by the retrieved examples.