Datastore Composition in k-NN Language Models
The datastore for a k-NN Language Model is a collection of key-value tuples. Each tuple links a context representation (key) to its corresponding ground-truth next token (value). A set of such tuples is represented as . In this structure, each key is the final hidden state vector from the LLM's Transformer at a specific position , and the value is the actual token that follows in the sequence. The datastore is populated by processing a large training corpus and collecting these pairs for every token position.
0
1
References
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Computational Challenge of Large-Scale k-NN Datastores
Datastore Composition in k-NN Language Models
Consider two language models that use an external datastore of (context -> next word) examples to help generate text.
- Model X populates its datastore only with examples from the specific document it is currently generating.
- Model Y's datastore is pre-filled with millions of examples from a vast and diverse library of texts before it begins generating any new document.
When asked to complete a sentence about a niche historical fact not mentioned earlier in the current document, which model is more likely to perform better and why?
Designing a Memory-Augmented Legal AI
Trade-offs in k-NN Datastore Population
Learn After
A language model is processing the text 'The quick brown fox jumps'. To build a datastore for a nearest-neighbor component, it must create a key-value pair
(z, w)for the context at each token. For the token 'brown', what would constitute the correct key-value pair?Correcting Datastore Construction in a k-NN Language Model
Analyzing Datastore Key-Value Pair Creation