Performance Metric for Instruction-Tuned LLMs
The effectiveness of an instruction-fine-tuned model, formally represented as , is assessed using a performance metric. This evaluation can be written as or, in a more concise form, as , where , , and represent the instruction, input, and output, respectively.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Two Levels of Generalization in Instruction-Tuned LLMs
Complexity of Generalization due to Instruction and Input Variation
A development team fine-tunes a large language model to be a helpful assistant for summarizing legal documents. They use a large dataset of legal texts and their corresponding summaries. After deployment, they observe the following:
- The model performs exceptionally well when asked to summarize new, unseen legal documents (e.g., contracts, court rulings).
- However, when users give it slightly different instructions, such as 'Explain this legal clause in simple terms,' 'Extract the key dates
Evaluating Fine-Tuning Strategies for Generalization
Performance Metric for Instruction-Tuned LLMs
Formal Representation of an Instruction-Tuned LLM
A large language model has been fine-tuned on a variety of instructional tasks. Match each of the following performance observations with the specific type of generalization challenge it represents.
Learn After
Critique of a Unified Performance Metric for AI
A researcher is evaluating an instruction-tuned language model using a performance metric denoted as P(c, z, y). The model is given the following:
- Instruction (c): "Summarize the following text into a single sentence."
- Input (z): "The sun is a star at the center of the Solar System. It is a nearly perfect sphere of hot plasma, with internal convective motion that generates a magnetic field via a dynamo process. It is by far the most important source of energy for life on Earth."
Analyzing Model Performance Components