Formal Definition of Intra-Task Generalization
Intra-task generalization refers to a model's ability to perform well on new, unseen inputs for a specific, fixed task. Formally, for a given task defined by instruction , a model is said to generalize if its average performance across all new inputs in a set exceeds a certain performance threshold . This demonstrates that the model has learned the underlying task rather than just memorizing training examples.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
LLM Generalization Evaluation
Definition of Intra-Task Generalization
Formal Definition of Intra-Task Generalization
An AI team fine-tunes a language model exclusively on a dataset for a single task: translating English legal documents into French. The model is then evaluated on two test sets.
- Test Set A: A new, unseen collection of English legal documents to be translated into French.
- Test Set B: A collection of diverse tasks, such as writing Python code, composing poetry, and summarizing news articles.
The model performs very well on Test Set A but performs poorly on Test Set B. What does this evaluation reveal about the model's generalization abilities?
Analyzing LLM Performance
Formula for Generalization Across Tasks
Learn After
An AI research team fine-tunes a large language model exclusively on the task of translating English sentences into formal logic. After training on a large dataset, they evaluate its performance. According to the formal definition of intra-task generalization, which of the following outcomes would best demonstrate that the model has successfully generalized for this specific task?
Evaluating Chatbot Generalization
A language model is trained to summarize news articles. When tested on the exact same set of articles used during its training, it achieves 100% accuracy. According to the formal definition, this result is sufficient to demonstrate strong intra-task generalization.
Formula for Generalization Within a Task