Google

To enhance the attention mechanism for a given query $$\mathbf{q}_i$$, language models aim to utilize both the immediate local memory, such as the standard Key-Value (KV) cache of recent tokens denoted as $$\mathrm{Mem}$$, and the long-term memory retrieved via $$k$$-nearest neighbors, denoted as $$\mathrm{Mem}_{k\mathrm{nn}}$$. Strategies to integrate these two sources of information include combining them to form a single, unified KV cache, $$[\mathrm{Mem}, \mathrm{Mem}_{k\mathrm{nn}}]$$, and applying standard QKV attention, or using $$\mathrm{Mem}$$ and $$\mathrm{Mem}_{k\mathrm{nn}}$$ in separate, distinct attention steps.

Integrating k-NN Memory with Local Memory in Attention

One straightforward method for integrating retrieved $$k$$-NN memory is to concatenate it with the local memory. In this approach, the local memory ($$\mathrm{Mem}$$) and the $$k$$-NN memory ($$\mathrm{Mem}_{k\mathrm{nn}}$$) are combined to form a single, larger Key-Value cache, represented as $$[\mathrm{Mem}, \mathrm{Mem}_{k\mathrm{nn}}]$$. The model then performs a standard query-key-value attention operation on this unified cache for a given query $$\mathbf{q}_i$$.

Combined KV Cache for k-NN and Local Memory

k-NN search augmented attention is an architecture that processes a query vector, `qi`, through two parallel computational streams. The first stream computes standard attention over a local memory, such as the KV cache, to capture immediate context. Concurrently, the second stream uses `qi` to perform a k-NN search on an external datastore, retrieving the k nearest neighbors. A separate attention calculation is then performed over these retrieved neighbors. Finally, the outputs from these two streams are integrated to produce a final result. This integration can occur at different levels, for instance, by combining the attention output vectors directly or by converting each stream's output into a probability distribution and then merging them.

k-NN Search Augmented Attention

Imagine you are designing a large language model to act as a specialized legal assistant. The model must be able to both engage in a coherent, multi-turn dialogue about a user's current legal query and accurately pull in specific, relevant precedents from a vast database of historical case law. The model's attention mechanism has access to two sources of information: 1) a 'local memory' containing the immediate context of the current conversation, and 2) a 'retrieved memory' containing the top 'k' most relevant text passages from the case law database, found using a search process. Propose two distinct architectural strategies for how the model's attention mechanism could combine these two memory sources to generate its next response. Analyze the potential trade-offs of each strategy specifically for this legal assistant scenario, and conclude by recommending one strategy over the other, justifying why it is better suited for balancing conversational coherence with factual accuracy.

Optimizing Attention for a Specialized Chatbot

A team is designing a language model for a legal chatbot. The model must be able to follow the immediate flow of a user's query while also referencing specific, relevant legal precedents from a massive, static database. Which of the following approaches for the model's attention mechanism best addresses this dual requirement?

Based on the described issues, analyze the limitations of the chatbot's current memory system. Explain how the two distinct problems correspond to failures in handling different types of contextual information.

Diagnosing Memory Deficiencies in a Chatbot

When incorporating both a local memory, $$\mathrm{Mem}$$, and a retrieved long-term memory, $$\mathrm{Mem}_{k\mathrm{nn}}$$, into a language model, one architectural approach is to process them in separate attention steps. As exemplified by the model developed by Wu et al. (2021), the outputs from the local attention mechanism and the external $$k$$-NN attention mechanism can then be linearly combined to produce the final representation.

Learn Before

Related