A team is developing a prompt for a chatbot that must answer questions based on a specific technical manual. They test two prompts on a set of 50 questions:

Google

An alternative approach to measuring a prompt's effectiveness is to use the log-likelihood of the output generated by a Large Language Model. This metric serves as a quantitative indicator of the prompt's quality.

Using Log-Likelihood to Evaluate Prompts

A developer is testing two prompts for a text summarization task. 

- **Prompt 1** results in a summary with a very high log-likelihood score from the model, but human evaluators rate the summary as 'poor' because it misses key points.
- **Prompt 2** results in a summary with a lower log-likelihood score, but human evaluators rate the summary as 'excellent' because it accurately captures all key points.

Based on this scenario, what is the most accurate conclusion about evaluating these prompts?

Prompt Evaluation for a Factual Task

A team is optimizing a prompt for a text classification task. They are considering two different methods for evaluating which of their candidate prompts is best:

- **Method A:** For a given text input and its correct label, they measure the model's calculated probability of generating that specific correct label. A higher probability is considered indicative of a better prompt.
- **Method B:** They use the prompt to generate a label for a large set of text inputs. They then calculate the overall accuracy by comparing the model-generated labels to the known correct labels. A higher accuracy score is considered indicative of a better prompt.

Analyze the primary advantages and disadvantages of using Method A compared to Method B for evaluating prompt effectiveness in this scenario.

Learn Before

Related