k-NN LM Interpolation Formula
The final probability distribution in a -nearest neighbors (-NN) language model is computed using a linear function that interpolates between the retrieval-based distribution, , and the base large language model (LLM) output distribution, . This interpolation uses a coefficient to balance the two components, represented by the formula: , where is the query's hidden state.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
k-NN LM Interpolation Formula
A developer is working with a language model that generates text by combining its own internal predictions with a probability distribution derived from similar examples in a large external datastore. The developer observes that the model's output is often creative but frequently ignores the specific, factual information present in the retrieved examples. To make the model's output adhere more closely to the facts in the datastore, how should the developer adjust the interpolation coefficient that balances these two distributions?
Analyzing Extreme Interpolation Scenarios
Diagnosing a Retrieval-Augmented Chatbot
Learn After
Next Token Selection in k-NN Language Models
A language model's final probability for a word is determined by blending its own internal prediction with a prediction based on retrieved text examples. The formula used is:
Final_Prob = λ * Retrieved_Prob + (1 - λ) * Internal_Prob. In a scenario where the model's internal prediction for the next word is 'innovative', but the most frequent word in similar retrieved examples is 'creative', how would the value of the coefficientλinfluence the outcome?Analyzing Component Influence in a k-NN Language Model
In a language model that combines its own predictions with information from retrieved examples using the formula
Final_Prob = λ * Retrieved_Prob + (1 - λ) * Base_LM_Prob, setting the coefficientλto 0 results in the final prediction being determined entirely by the retrieved examples.