Choosing the Bottleneck in a Fraud-Review Pipeline
Scenario: A payment platform uses a two-step system to catch suspicious purchases. One module first marks transactions that look unusual. A second module then decides whether each marked transaction is truly fraudulent.
An evaluation shows that the first module is still much worse than experienced human reviewers, while the second module is already nearly as good as those reviewers.
Question: Using the idea of comparing each pipeline stage with human performance, which module should the team improve first, and what is the reason?
Sample answer: The first module should be the top priority. It is still far from the human benchmark, so there is more room for meaningful gains. The second module is already close to human performance, so additional work on it is less likely to produce large improvements.
Key points:
- Name the transaction-flagging module as the main target for improvement.
- Explain that being well below human performance suggests substantial room to improve.
- Explain that the fraud-decision module is already near human performance, so it is a lower-priority target.
Rubric: The response must identify the transaction-flagging module as the main improvement target. It must justify the choice by noting that a stage far from human performance offers a better opportunity for progress, while a stage already near human performance is usually harder to improve further.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
What is the best response when one stage in a multi-step system is far worse than expert performance?
True or False: If a subsystem performs much worse than expert-level performance on the same task, it is a strong candidate for targeted improvement.
If a pipeline stage is far from _____ performance, it is a strong candidate for improvement.
Match each model-status description to the right improvement priority.
Order the steps for using human-level comparisons to decide which subsystem to improve first.
Why should a subsystem far below expert-level performance usually receive top attention?
True or False: The best place to focus improvement is a module that already matches human performance closely.
When one module is far worse than human performance, you have a strong _____ to improve that module.
Match each model-performance scenario to the most appropriate takeaway.
Order the reasoning chain for choosing the weakest stage in a multi-step system to improve first.
Using a Human Benchmark to Choose Where to Improve a Model
Choosing the Bottleneck in a Fraud-Review Pipeline
When to prioritize improving one pipeline stage