Learn Before
Formula for Generalization Within a Task
The condition for a model to demonstrate generalization within a given task for a specific instruction is met if its average performance on a set of new inputs exceeds a minimum threshold . This is expressed mathematically by the formula:
where represents the set of new inputs, is a specific input from this set, and is the model's corresponding output.

0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
An AI research team fine-tunes a large language model exclusively on the task of translating English sentences into formal logic. After training on a large dataset, they evaluate its performance. According to the formal definition of intra-task generalization, which of the following outcomes would best demonstrate that the model has successfully generalized for this specific task?
Evaluating Chatbot Generalization
A language model is trained to summarize news articles. When tested on the exact same set of articles used during its training, it achieves 100% accuracy. According to the formal definition, this result is sufficient to demonstrate strong intra-task generalization.
Formula for Generalization Within a Task
Learn After
A research team is evaluating a language model's ability to generalize on the specific task of 'translating medical terminology from English to German'. The condition for successful generalization is met if the model's average performance on a set of new inputs exceeds a minimum threshold. This is represented by the formula: Where is the set of new inputs, is the performance score for a given input, and is the performance threshold.
The team tests the model on 5 new, unseen medical texts () and sets the minimum performance threshold at . The individual performance scores for the 5 texts are: [0.90, 0.95, 0.70, 0.80, 0.90].
Based on this data and the provided formula, which conclusion is correct?
The condition for a model to demonstrate intra-task generalization is expressed by the formula: Match each component of this formula to its correct description.
Evaluating Generalization Performance in a Real-World Scenario