Learn Before
Role of Internal State in Datastore Search
A language model is enhanced with a large datastore containing numerous text examples. To help predict the next word, the model uses its internal representation of the current text sequence to search this datastore. Explain the specific role of this internal representation in the search process and what specific information is retrieved from the 'k' most similar examples that are found.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Using Reference Tokens to Define a Vocabulary Distribution in k-NN LM
Role of Internal State in Datastore Search
A language model enhanced with a nearest-neighbor mechanism needs to find relevant information from its external datastore to help predict the next word. Arrange the following steps in the correct chronological order to describe how the model retrieves this information.
A language model enhanced with a nearest-neighbor search mechanism is generating text. The model's current internal state, representing the prefix 'The scientist made a groundbreaking...', is used as a query to search an external datastore. The datastore contains pairs of (context representation, associated word). If the search retrieves the three words 'discovery', 'advance', and 'finding' as reference tokens, which statement most accurately describes how these specific words were selected?