Case Study

Choosing the Bottleneck in a Fraud-Review Pipeline

Scenario: A payment platform uses a two-step system to catch suspicious purchases. One module first marks transactions that look unusual. A second module then decides whether each marked transaction is truly fraudulent.

An evaluation shows that the first module is still much worse than experienced human reviewers, while the second module is already nearly as good as those reviewers.

Question: Using the idea of comparing each pipeline stage with human performance, which module should the team improve first, and what is the reason?

Sample answer: The first module should be the top priority. It is still far from the human benchmark, so there is more room for meaningful gains. The second module is already close to human performance, so additional work on it is less likely to produce large improvements.

Key points:

  • Name the transaction-flagging module as the main target for improvement.
  • Explain that being well below human performance suggests substantial room to improve.
  • Explain that the fraud-decision module is already near human performance, so it is a lower-priority target.

Rubric: The response must identify the transaction-flagging module as the main improvement target. It must justify the choice by noting that a stage far from human performance offers a better opportunity for progress, while a stage already near human performance is usually harder to improve further.

0

1

Updated 2026-08-12

Contributors are:

Who are from:

Tags

Machine Learning

Deep Learning

Supervised Learning

Dive into Deep Learning @ D2L

Data Science

Machine Learning Strategy

Machine Learning Yearning @ DeepLearning.AI