Google

The datastore for a k-NN Language Model is a collection of key-value tuples. Each tuple links a context representation (key) to its corresponding ground-truth next token (value). A set of such tuples is represented as $\{(\mathbf{Z}_1, w_1), ..., (\mathbf{Z}_k, w_k)\}$. In this structure, each key $\mathbf{Z}_i$ is the final hidden state vector from the LLM's Transformer at a specific position $i$, and the value $w_i$ is the actual token that follows in the sequence. The datastore is populated by processing a large training corpus and collecting these $(\mathbf{Z}_i, w_i)$ pairs for every token position.

Datastore Composition in k-NN Language Models

A language model is processing the text 'The quick brown fox jumps'. To build a datastore for a nearest-neighbor component, it must create a key-value pair `(z, w)` for the context at each token. For the token 'brown', what would constitute the correct key-value pair?

A developer is creating a datastore for a k-nearest-neighbor language model by processing the sentence 'The cat sat on the mat'. For the token 'sat', they create a key-value pair where the key is the model's final hidden state vector at the position of 'sat', and the value is the token 'sat' itself. Analyze why this construction is incorrect and describe the correct value that should be paired with this key.

Correcting Datastore Construction in a k-NN Language Model

Based on the provided scenario, analyze the engineer's method for creating a key-value pair. Identify any errors in their approach and explain the correct procedure, justifying your reasoning based on the function of the datastore.

Learn Before

Related