Extending k-NN Datastore Context with a Training Dataset
An alternative to using only the current sequence for context is to populate the k-NN datastore with key-value pairs from a larger collection of sequences, such as an entire training dataset. This approach enables a Large Language Model to leverage a more generalized context for making predictions.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Extending k-NN Datastore Context with a Training Dataset
A language model equipped with a k-NN based memory is processing the sentence: 'The quick brown fox'. The model processes the sentence one word at a time, from left to right. When the model is about to predict the next word after 'brown', which of the following best describes the contents of its memory datastore according to the standard sequential population strategy?
Implications of Sequential Datastore Population
A language model is equipped with a k-NN based memory that is populated using the standard strategy of storing the history of the current sequence. The model begins processing a new sequence with an empty datastore. Arrange the following events in the correct chronological order as the model processes the first two tokens.
Learn After
Computational Challenge of Large-Scale k-NN Datastores
Datastore Composition in k-NN Language Models
Consider two language models that use an external datastore of (context -> next word) examples to help generate text.
- Model X populates its datastore only with examples from the specific document it is currently generating.
- Model Y's datastore is pre-filled with millions of examples from a vast and diverse library of texts before it begins generating any new document.
When asked to complete a sentence about a niche historical fact not mentioned earlier in the current document, which model is more likely to perform better and why?
Designing a Memory-Augmented Legal AI
Trade-offs in k-NN Datastore Population