Language Model Performance Diagnosis
Based on the following scenario, identify the specific type of performance issue the model is exhibiting and explain the reasoning for your diagnosis.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Computing Sciences
Foundations of Large Language Models Course
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Language Model Performance Diagnosis
A development team is analyzing the performance of their language model. For a set of test prompts, they take the top 5 responses generated by their model and have a significantly more powerful, 'oracle' model select the best response from that list. They find that the average quality score of their model's original top-ranked response is 70%, while the average quality score of the response selected by the oracle model is 95%. What does this large performance gap most strongly suggest about the primary limitation of the team's model?
Interpreting Model Diagnostic Results
Search Errors in LLMs