Implications of Sequential Datastore Population
A language model is designed to populate its memory datastore by adding a new key-value pair for every token it processes in the current input sequence. If this model is used to process a very long document, what is a major challenge this population strategy will face regarding performance as it nears the end of the document? Explain the reason for this challenge.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Extending k-NN Datastore Context with a Training Dataset
A language model equipped with a k-NN based memory is processing the sentence: 'The quick brown fox'. The model processes the sentence one word at a time, from left to right. When the model is about to predict the next word after 'brown', which of the following best describes the contents of its memory datastore according to the standard sequential population strategy?
Implications of Sequential Datastore Population
A language model is equipped with a k-NN based memory that is populated using the standard strategy of storing the history of the current sequence. The model begins processing a new sequence with an empty datastore. Arrange the following events in the correct chronological order as the model processes the first two tokens.