Architecture Decision Memo: Unifying Vector-DB RAG and k-NN LM for a Global Policy Assistant
You are leading an architecture review for an internal “Policy & Procedures” assistant used by Legal, HR, and Finance. The assistant must (1) answer questions with citations to the exact policy passages used, (2) stay current as policies change weekly, and (3) support multi-turn chats where later turns depend on earlier answers. Your team proposes a single vector database to support both: (a) a RAG pipeline that retrieves the top-k relevant policy snippets to include in the prompt, and (b) a k-NN language modeling (k-NN LM) component that retrieves nearest-neighbor hidden-state entries to influence next-token prediction during generation.
Write a decision memo that evaluates whether using one shared vector database for both retrieval paths is a good idea. In your answer, you must:
- Explain how text retrieval for RAG and k-NN LM differ in what they store as vectors, what the query vector represents, and what the retrieved items are used for.
- Propose a concrete design (one shared store or two separate stores) and justify it using the grounding/citation requirement and the need for rapid updates.
- Identify at least two failure modes that could occur if the retrieval design is wrong (e.g., correct-sounding but ungrounded answers, stale policy usage, irrelevant retrieval due to embedding mismatch), and describe how you would detect/mitigate them.
Assume you cannot fine-tune the base LLM and must rely on retrieval-time mechanisms.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Ch.3 Prompting - Foundations of Large Language Models
Ch.5 Inference - Foundations of Large Language Models
Related
Augmented Input Formula in RAG
k-NN Language Modeling (k-NN LM)
Example of Retrieval-Augmented Generation
RAG for Fact-Intensive Tasks
Key Steps in Retrieval-Augmented Generation (RAG)
Comparison of RAG and Fine-Tuning for LLM Adaptation
Training-Free Nature of Standard RAG
Potential for RAG Framework Improvement
Comparison of Execution Timing in Tool Use and RAG
Grounding LLM Responses with External Sources in RAG
Addressing LLM Knowledge Limitations with RAG
A company has built a customer support chatbot using a large language model. They notice that while the chatbot is excellent at general conversation, it frequently provides inaccurate information about product specifications that were updated last month, after the model's training data was finalized. Which of the following approaches best describes a method to ground the model's responses in the most current, verifiable information for each user query?
A user submits a query to a system designed to provide factually accurate answers by dynamically incorporating external knowledge. Arrange the following steps to correctly represent the operational flow of this system.
Retrieval-Augmented Generation Process
Diagnosing a Knowledge-Augmented System Failure
Design Review: Choosing Between RAG and k-NN LM for a Regulated Support Assistant
Post-Incident Analysis: Why a RAG Assistant Hallucinated Despite “Having the Docs”
Architecture Decision Memo: Unifying Vector-DB RAG and k-NN LM for a Global Policy Assistant
Case Review: Diagnosing Conflicting Answers in a Hybrid Retrieval System
Case Study: Debugging a RAG Assistant with a Vector DB and a k-NN LM Memory
Case Study: Root-Cause Analysis of “Correct Source, Wrong Answer” in a RAG + k-NN LM Assistant
You are reviewing two proposed designs for an inte...
Your team is building an internal “Release Notes Q...
You’re on-call for an internal engineering assista...
You’re designing an internal LLM assistant for a c...
RAG as Problem Decomposition
Inference Architecture of k-NN Language Models
Next-Token Prediction with External Memory
A language model is enhanced by searching a large datastore of past internal states and their corresponding next words. When the model generates a new word, it finds the 'k' most similar past states from the datastore and uses their associated next words to adjust its prediction. What is the key principle that makes this technique effective?
Foundational Principle of k-NN Language Modeling
A language model is designed to improve its next-word predictions by consulting a large external database of past contexts. Arrange the following steps to accurately describe how this model generates its final output after receiving an input.
A language model is designed to enhance its next-token prediction by referencing a large external datastore of context representations and their corresponding subsequent tokens. During generation, for a given input, the model identifies the 'k' most similar context representations from this datastore. Which of the following best describes how this information is integrated to produce the final prediction?
You’re on-call for an internal engineering assista...
You are reviewing two proposed designs for an inte...
Your team is building an internal “Release Notes Q...
You’re designing an internal LLM assistant for a c...
Design Review: Choosing Between RAG and k-NN LM for a Regulated Support Assistant
Post-Incident Analysis: Why a RAG Assistant Hallucinated Despite “Having the Docs”
Architecture Decision Memo: Unifying Vector-DB RAG and k-NN LM for a Global Policy Assistant
Case Study: Root-Cause Analysis of “Correct Source, Wrong Answer” in a RAG + k-NN LM Assistant
Case Study: Debugging a RAG Assistant with a Vector DB and a k-NN LM Memory
Case Review: Diagnosing Conflicting Answers in a Hybrid Retrieval System
k-NN Memory Retrieval
Pre-indexing k-NN Datastores for Efficient Retrieval
An e-commerce company has converted its catalog of 10 million product descriptions into high-dimensional numerical vectors. They want to build a search feature where a user's text query is also converted into a vector, and the system must rapidly return the top 10 products with the most similar description vectors. Which data storage solution is best suited for this specific task?
Architectural Review for a Similarity Search System
Choosing the Right Database for Similarity Search
You’re on-call for an internal engineering assista...
You are reviewing two proposed designs for an inte...
Your team is building an internal “Release Notes Q...
You’re designing an internal LLM assistant for a c...
Design Review: Choosing Between RAG and k-NN LM for a Regulated Support Assistant
Post-Incident Analysis: Why a RAG Assistant Hallucinated Despite “Having the Docs”
Architecture Decision Memo: Unifying Vector-DB RAG and k-NN LM for a Global Policy Assistant
Case Study: Root-Cause Analysis of “Correct Source, Wrong Answer” in a RAG + k-NN LM Assistant
Case Study: Debugging a RAG Assistant with a Vector DB and a k-NN LM Memory
Case Review: Diagnosing Conflicting Answers in a Hybrid Retrieval System
A company develops an AI chatbot to answer employee questions about its internal HR policies, which are stored in a document database. Employees report that the chatbot sometimes provides answers based on outdated policies, even though the document database is updated daily. Which of the following is the most likely reason for this issue?
Evaluating Response Grounding in a RAG System
The Role of Grounding in RAG Systems
You’re on-call for an internal engineering assista...
You are reviewing two proposed designs for an inte...
Your team is building an internal “Release Notes Q...
You’re designing an internal LLM assistant for a c...
Design Review: Choosing Between RAG and k-NN LM for a Regulated Support Assistant
Post-Incident Analysis: Why a RAG Assistant Hallucinated Despite “Having the Docs”
Architecture Decision Memo: Unifying Vector-DB RAG and k-NN LM for a Global Policy Assistant
Case Study: Root-Cause Analysis of “Correct Source, Wrong Answer” in a RAG + k-NN LM Assistant
Case Study: Debugging a RAG Assistant with a Vector DB and a k-NN LM Memory
Case Review: Diagnosing Conflicting Answers in a Hybrid Retrieval System
Implementing RAG Retrieval with Vector Databases
An automated system is designed to answer user questions. Its first step is to search a large document library to find the most relevant texts related to the user's query. The system will then use only these retrieved texts to generate a final answer. A user asks: 'What are the primary health benefits of a Mediterranean diet?' Which of the following sets of retrieved documents would be the most effective for the system's next step?
Using Off-the-Shelf Information Retrieval Systems for RAG
Diagnosing a Flawed Generative Response
Evaluating Retrieval Relevance
You’re on-call for an internal engineering assista...
You are reviewing two proposed designs for an inte...
Your team is building an internal “Release Notes Q...
You’re designing an internal LLM assistant for a c...
Design Review: Choosing Between RAG and k-NN LM for a Regulated Support Assistant
Post-Incident Analysis: Why a RAG Assistant Hallucinated Despite “Having the Docs”
Architecture Decision Memo: Unifying Vector-DB RAG and k-NN LM for a Global Policy Assistant
Case Study: Root-Cause Analysis of “Correct Source, Wrong Answer” in a RAG + k-NN LM Assistant
Case Study: Debugging a RAG Assistant with a Vector DB and a k-NN LM Memory
Case Review: Diagnosing Conflicting Answers in a Hybrid Retrieval System