Case Review: Preventing Incorrect Refund Commitments in an LLM + Payments API Assistant
You are reviewing an internal pilot of an LLM-powered customer support assistant for a subscription product. The assistant can call two external APIs:
get_invoice(customer_id, invoice_id)→ returns line items, taxes, discounts, currency, and current payment status.create_refund(invoice_id, amount, currency, reason)→ executes a refund immediately and returns a refund confirmation ID.
Incident: A customer asked, “I was double-charged on invoice INV-8841—refund the extra charge.” The assistant responded confidently: “You were charged twice; I’ve refunded $49.99,” and then called create_refund(INV-8841, 49.99, "USD", "duplicate charge"). Later, finance found the invoice was in EUR, the ‘double charge’ was actually an authorization + capture, and the correct action was to provide an explanation (no refund). The team wants a redesign that (1) minimizes extra model calls/latency, (2) reduces the chance of executing an incorrect refund, and (3) still uses the LLM to handle ambiguous customer language.
As the reviewer, propose a single end-to-end workflow (not a list of unrelated tips) that integrates: (a) a deliberate-then-generate step, (b) a predict-then-verify mechanism with an explicit verifier, (c) self-reflection to catch overconfident claims, and (d) safe external API tool use. Your answer must specify where in the flow the model generates candidates, what the verifier checks (outcome vs. step-level), what evidence must be pulled from get_invoice, and the exact gating rule that prevents create_refund from being called when uncertainty or mismatches (e.g., currency/status) are detected.
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Ch.5 Inference - Foundations of Large Language Models
Related
Comparison of Execution Timing in Tool Use and RAG
Limitation of Pre-trained LLMs in Tool Use
Using Web Search as an External Tool for LLMs
Application of LLMs to Mathematical Problems
A development team is building a chatbot for an airline. The chatbot must be able to answer user questions like, 'What is the status of flight UA456?' and 'Are there any business class seats available on the 10 AM flight to London tomorrow?'. The airline's flight data is stored in a private, real-time database. Which of the following represents the most effective and reliable approach for the team to implement these features?
A user asks a Large Language Model, 'What is the capital of Brazil and what is the current time there?'. The model has access to an external tool
get_current_time(city). Arrange the following steps in the logical order the model would follow to answer the user's request.Troubleshooting an LLM-Powered Sales Assistant
Designing a Reliable LLM Workflow for Real-Time Decisions
Design Review: Combining Tool Use, DTG, and Predict-then-Verify for a High-Stakes API Workflow
Post-Incident Analysis: Preventing Confidently Wrong API-Backed Answers
Case Review: Preventing Incorrect Refund Commitments in an LLM + Payments API Assistant
Case Study: Shipping a Tool-Using LLM Assistant with Built-In Verification Under Latency Constraints
Case Study: Preventing Hallucinated Compliance Claims in an API-Enabled LLM for Vendor Risk Reviews
You are reviewing a proposed architecture for an i...
In an LLM-based customer support assistant, the mo...
You’re designing an internal LLM assistant for a f...
You’re leading an internal rollout of an LLM assis...
Methods for Activating Self-Reflection in LLMs
An AI model is asked, 'What is the approximate distance from the Earth to the Moon?' It provides two consecutive responses:
- Response 1: 'The distance from the Earth to the Moon is about 238,900 kilometers.'
- Response 2: 'Upon review, my previous answer was imprecise. The distance is in miles, not kilometers. The correct average distance is approximately 238,900 miles, which is about 384,400 kilometers. Stating the unit correctly is crucial for accuracy.'
Which of the following b
Evaluating AI Response Quality
Mechanism of AI Self-Correction
You are reviewing a proposed architecture for an i...
You’re designing an internal LLM assistant for a f...
You’re leading an internal rollout of an LLM assis...
In an LLM-based customer support assistant, the mo...
Design Review: Combining Tool Use, DTG, and Predict-then-Verify for a High-Stakes API Workflow
Designing a Reliable LLM Workflow for Real-Time Decisions
Post-Incident Analysis: Preventing Confidently Wrong API-Backed Answers
Case Study: Shipping a Tool-Using LLM Assistant with Built-In Verification Under Latency Constraints
Case Review: Preventing Incorrect Refund Commitments in an LLM + Payments API Assistant
Case Study: Preventing Hallucinated Compliance Claims in an API-Enabled LLM for Vendor Risk Reviews
Limitation of the Deliberate-then-Generate (DTG) Method
Comparison of Iterative vs. Non-Iterative Prompting Methods
Instructional Component of the DTG Prompt Template for Translation Refinement
Integration of Feedback and Refinement in the DTG Method
A developer is using a Large Language Model to refine a technical summary. They want the model to first identify any factual inaccuracies or unclear statements in the original text and then, based on that analysis, produce a corrected and more coherent version. Which of the following approaches correctly implements the 'Deliberate-then-Generate' method for this task?
Input Structure of the DTG Prompt for Chinese-to-English Translation
Challenge of LLM-Based Error Identification in Translation
A developer is designing a workflow to refine user-generated reports using a Large Language Model. The primary goal is to ensure the model first analyzes potential issues (e.g., ambiguity, factual errors) before rewriting the report, all while minimizing the number of interactions with the model. Which of the following prompt structures best represents the 'Deliberate-then-Generate' method for this task?
Analysis of a Translation Refinement Process
You are reviewing a proposed architecture for an i...
You’re designing an internal LLM assistant for a f...
You’re leading an internal rollout of an LLM assis...
In an LLM-based customer support assistant, the mo...
Design Review: Combining Tool Use, DTG, and Predict-then-Verify for a High-Stakes API Workflow
Designing a Reliable LLM Workflow for Real-Time Decisions
Post-Incident Analysis: Preventing Confidently Wrong API-Backed Answers
Case Study: Shipping a Tool-Using LLM Assistant with Built-In Verification Under Latency Constraints
Case Review: Preventing Incorrect Refund Commitments in an LLM + Payments API Assistant
Case Study: Preventing Hallucinated Compliance Claims in an API-Enabled LLM for Vendor Risk Reviews