Example of Performance Estimation in Prompt Optimization

An engineering team is building an automated system to discover the most effective instructions for a language model to generate Python code. The system generates thousands of instruction variations and needs a way to determine which one is the best. The team observes that the system often selects instructions that produce code that looks syntactically correct but fails to run due to subtle errors. Which part of this automated discovery process is most likely the source of this issue?

Designing a Performance Metric for Summarization Prompts

Critiquing a Flawed Prompt Evaluation Method