Learn Before
Critique of an AI Performance Metric
A company develops an AI model to assist software developers by automatically generating code. To measure the model's performance, the company decides to use 'lines of code generated per hour' as the primary success metric. The model is then optimized to maximize this metric. Critically evaluate this choice of metric. In your response, analyze the potential unintended consequences of this optimization strategy on the quality and utility of the generated code, and propose a more effective evaluation framework.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Evaluation in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A company develops a large language model to summarize news articles. To evaluate its performance, they decide to reward the model for producing summaries that have a high percentage of word overlap with the original article's headline, believing this indicates the summary has captured the main point. After training, they find the model produces summaries that are often just slight rephrasings of the headline, failing to include other crucial information from the article. Which of the following principles best explains this unintended outcome?
Critique of an AI Performance Metric
AI Content Strategy Analysis