Formal Definition of Intra-Task Generalization
Intra-task generalization refers to a model's ability to perform well on new, unseen inputs for a specific, fixed task. Formally, for a given task defined by instruction , a model is said to generalize if its average performance across all new inputs in a set exceeds a certain performance threshold . This demonstrates that the model has learned the underlying task rather than just memorizing training examples.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
LLM Generalization Evaluation
Formal Definition of Intra-Task Generalization
An AI team fine-tunes a language model exclusively on a dataset for a single task: translating English legal documents into French. The model is then evaluated on two test sets.
- Test Set A: A new, unseen collection of English legal documents to be translated into French.
- Test Set B: A collection of diverse tasks, such as writing Python code, composing poetry, and summarizing news articles.
The model performs very well on Test Set A but performs poorly on Test Set B. What does this evaluation reveal about the model's generalization abilities?
Analyzing LLM Performance
Formula for Generalization Across Tasks
Informal Definition of Intra-Task Generalization
Learn After
An AI research team fine-tunes a large language model exclusively on the task of translating English sentences into formal logic. After training on a large dataset, they evaluate its performance. According to the formal definition of intra-task generalization, which of the following outcomes would best demonstrate that the model has successfully generalized for this specific task?
Evaluating Chatbot Generalization
A language model is trained to summarize news articles. When tested on the exact same set of articles used during its training, it achieves 100% accuracy. According to the formal definition, this result is sufficient to demonstrate strong intra-task generalization.
Formula for Generalization Within a Task