Google

In the $$k$$-nearest neighbors ($$k$$-NN) retrieval process, a datastore of key-value pairs, $$\mathbf{k}_j$$ and $$\mathbf{v}_j$$, is maintained, often within a vector database. For a given query $$\mathbf{q}_i$$, the system identifies its $$k$$ nearest neighbors from the set of keys by conceptually expanding a sphere centered at $$\mathbf{q}_i$$ until it encompasses exactly $$k$$ data points in $$\mathbf{k}_j$$. This search results in a retrieved long-term memory set, denoted as $$\mathrm{Mem}_{k\mathrm{nn}}$$, which contains the $$k$$ keys along with their corresponding values.

k-NN Memory Retrieval

k-NN search augmented attention is an architecture that processes a query vector, `qi`, through two parallel computational streams. The first stream computes standard attention over a local memory, such as the KV cache, to capture immediate context. Concurrently, the second stream uses `qi` to perform a k-NN search on an external datastore, retrieving the k nearest neighbors. A separate attention calculation is then performed over these retrieved neighbors. Finally, the outputs from these two streams are integrated to produce a final result. This integration can occur at different levels, for instance, by combining the attention output vectors directly or by converting each stream's output into a probability distribution and then merging them.

k-NN Search Augmented Attention

In a memory retrieval system, a query is compared against a large datastore of key-value pairs to find the 'k' most similar keys. The corresponding key-value pairs are then returned. What is the primary effect of increasing the value of 'k'?

Analyze the following scenario and identify the single parameter of the retrieval process that is most likely causing the described issue. Propose a specific adjustment to this parameter and explain your reasoning.

Optimizing a Chatbot's Retrieval System

A system is designed to retrieve information from a datastore of key-value pairs using a nearest-neighbor approach. Arrange the following steps of this retrieval process in the correct logical sequence.

Learn Before

Related