Case Study

Evaluating Prompt Search Strategies

A startup is optimizing a prompt for a chatbot that handles a single, well-defined task: processing product returns. They have a limited budget for computation and need to find a good prompt quickly. They are considering two different search strategies:

  • Strategy A: Systematically generate and test 10,000 distinct prompt variations against a validation dataset, then select the single highest-performing prompt.
  • Strategy B: Start with 5 initial prompts. In each step, test the current set, discard the poor performers, and generate a few new variations based on the most successful ones. Repeat this process until the performance score stops improving.

Based on the startup's constraints, which strategy would you recommend? Justify your choice by evaluating the primary trade-off between these two approaches.

0

1

Updated 2025-10-02

Contributors are:

Who are from:

Tags

Ch.3 Prompting - Foundations of Large Language Models

Foundations of Large Language Models

Foundations of Large Language Models Course

Computing Sciences

Evaluation in Bloom's Taxonomy

Cognitive Psychology

Psychology

Social Science

Empirical Science

Science