Analyzing Extreme Interpolation Scenarios
A language model's final output is determined by blending its own internal predictions with information retrieved from an external datastore. This blending is controlled by a single coefficient. Analyze the distinct characteristics of the model's generated text under two extreme scenarios: 1) when the coefficient is set to give zero weight to the external datastore, and 2) when the coefficient is set to give all the weight to the external datastore.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
k-NN LM Interpolation Formula
A developer is working with a language model that generates text by combining its own internal predictions with a probability distribution derived from similar examples in a large external datastore. The developer observes that the model's output is often creative but frequently ignores the specific, factual information present in the retrieved examples. To make the model's output adhere more closely to the facts in the datastore, how should the developer adjust the interpolation coefficient that balances these two distributions?
Analyzing Extreme Interpolation Scenarios
Diagnosing a Retrieval-Augmented Chatbot