Stabilizing an LLM Workflow for Multi-Step Policy Compliance Decisions
You are deploying an internal LLM assistant to help procurement analysts decide whether a proposed vendor contract requires (a) a standard review, (b) an enhanced review, or (c) an automatic rejection. The decision depends on multiple interdependent rules (e.g., data types handled, cross-border transfers, subcontractors, and exception clauses). In pilot testing, a single prompt that asks for the final decision often misses a key condition; a zero-shot “Let’s think step by step” prompt sometimes produces a long rationale but forgets to clearly state the final decision; and when you add a few demonstrations, the model becomes more consistent but still makes occasional early-step mistakes that cascade into the wrong outcome.
Design a prompting workflow (you may describe it as a sequence of LLM calls) that uses: (1) in-context learning via demonstrations, (2) explicit problem decomposition in a least-to-most progression, and (3) an iterative self-refinement loop. Your answer must explain how information flows from one step to the next, how you would prevent or detect cascading errors from early steps, and how you would ensure the model always outputs an unambiguous final decision label (a/b/c) even when using chain-of-thought style reasoning.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Ch.3 Prompting - Foundations of Large Language Models
Ch.5 Inference - Foundations of Large Language Models
Ch.1 Pre-training - Foundations of Large Language Models
Related
Application of COT Prompting on GSM8K Benchmark
Structuring Logical Reasoning Steps for Demonstrations
Zero-Shot Chain-of-Thought (COT) Prompting
Application of CoT to Algebraic Calculation Problems
Benefits of Chain-of-Thought (CoT) Prompting
Incomplete Answers from Zero-Shot CoT Prompts
Chain-of-Thought as a Search Process
Supervising Intermediate Reasoning Steps for LLM Alignment
Limitations of Simple Chain-of-Thought Prompting
Creating a CoT Prompt by Incorporating Reasoning Steps
Alternative Trigger Phrases for Zero-Shot CoT Prompting
Incomplete Answers as a Potential Issue in Zero-Shot CoT Prompting
A developer is trying to improve a language model's ability to solve multi-step word problems. They compare two prompting strategies.
Strategy 1: Provide the model with a new word problem and ask for the final answer directly.
Strategy 2: Provide the model with a new word problem, but first show it an example of a similar problem where the solution is explicitly broken down into logical, sequential steps before reaching the final conclusion.
Why is Strategy 2 generally more effective
Improving a Prompt for a Multi-Step Problem
Few-Shot Chain-of-Thought (CoT) Prompting
Practical Limitations of Chain-of-Thought Prompting
The primary benefit of a prompting technique that demonstrates a step-by-step reasoning process is that it permanently modifies the language model's internal weights, making it inherently better at solving similar problems in the future, even without the detailed prompt.
Designing a Prompting Workflow for a High-Stakes, Multi-Step Task
Choosing and Justifying a Prompting Strategy Under Context and Quality Constraints
Diagnosing and Redesigning a Prompting Approach for a Decomposed Workflow
Stabilizing an LLM Workflow for Multi-Step Policy Compliance Decisions
Debugging a Multi-Step LLM Workflow for Contract Clause Risk Triage
Designing a Robust Prompting Workflow for Multi-Step Root-Cause Analysis with Limited Examples
You’re building an internal LLM assistant to help ...
Your team is rolling out an internal LLM assistant...
You’re leading an internal enablement team buildin...
You’re building an internal LLM workflow to produc...
Example of One-Shot Chain-of-Thought (COT) Prompting
Problem-Solving Scenarios for Chain-of-Thought Prompting
Self-Consistency Method
Sub-problem Generation in Least-to-Most Prompting
Improving Least-to-Most Prompting with Advanced Techniques
Improving Problem Decomposition in Least-to-Most Prompting
An AI developer needs a large language model to solve a complex, multi-step logic puzzle that requires deducing a final answer from a series of interdependent clues. Initial attempts to solve the puzzle by providing the full puzzle and a few examples of other solved puzzles have consistently failed. Which of the following prompting strategies is the most effective next step, and why?
Analyzing a Problem-Solving Approach
A language model is tasked with solving the following logic puzzle: 'Sarah, David, and Emily are a doctor, a lawyer, and an engineer. The doctor is Emily's sister. David is not the lawyer.' To solve this complex problem, it is broken down into a series of simpler, sequential sub-problems. Arrange the following sub-problems in the correct logical order that builds towards the final solution.
Your team is rolling out an internal LLM assistant...
You’re building an internal LLM workflow to produc...
You’re building an internal LLM assistant to help ...
You’re leading an internal enablement team buildin...
Choosing and Justifying a Prompting Strategy Under Context and Quality Constraints
Designing a Prompting Workflow for a High-Stakes, Multi-Step Task
Diagnosing and Redesigning a Prompting Approach for a Decomposed Workflow
Stabilizing an LLM Workflow for Multi-Step Policy Compliance Decisions
Debugging a Multi-Step LLM Workflow for Contract Clause Risk Triage
Designing a Robust Prompting Workflow for Multi-Step Root-Cause Analysis with Limited Examples
Example of Final Problem Solving in Least-to-Most Prompting
Example of Self-Refinement in Machine Translation
Three-Step Framework for Self-Refinement in LLMs
Ideal Self-Refinement without Additional Training