Learn Before
Example of Performance Estimation in Prompt Optimization
A straightforward method for evaluating a prompt's performance involves inputting it to a Large Language Model and then measuring the model's performance on a dedicated validation set.
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Example of Performance Estimation in Prompt Optimization
An engineering team is building an automated system to discover the most effective instructions for a language model to generate Python code. The system generates thousands of instruction variations and needs a way to determine which one is the best. The team observes that the system often selects instructions that produce code that looks syntactically correct but fails to run due to subtle errors. Which part of this automated discovery process is most likely the source of this issue?
Designing a Performance Metric for Summarization Prompts
Critiquing a Flawed Prompt Evaluation Method
Learn After
A team is developing a system to classify customer feedback emails as 'Urgent' or 'Not Urgent'. They have designed two different sets of instructions for the language model to follow. To decide which set of instructions is more effective, what is the most appropriate next step?
You are tasked with evaluating a new prompt designed to make a language model summarize news articles. You have a dataset of 50 news articles, each with a corresponding human-written 'gold standard' summary. Arrange the following actions into the correct sequence to estimate the performance of your new prompt.
Discrepancy in Prompt Performance