1Cademy - Performance Metric for Instruction-Tuned LLMs

Instruction (c): &quot;Summarize the following text into a single sentence.&quot;
Input (z): &quot;The sun is a star at the center of the Solar System. It is a nearly perfect sphere of hot plasma, with internal convective motion that generates a magnetic field via a dynamo process. It is by far the most important source of energy for life on Earth.&quot;
Output (y): &quot;The sun, a central star in our solar system, is a vital energy source for Earth.&quot;

Learn Before

Generalization Challenges in Instruction Fine-Tuning

Formula

Performance Metric for Instruction-Tuned LLMs

The effectiveness of an instruction-fine-tuned model, formally represented as $\Pr(\mathbf{y}|\mathbf{c},\mathbf{z})$ , is assessed using a performance metric. This evaluation can be written as $\mathrm{Performance}(\Pr(\mathbf{y}|\mathbf{c},\mathbf{z}))$ or, in a more concise form, as $\mathrm{P}(\mathbf{c},\mathbf{z},\mathbf{y})$ , where $\mathbf{c}$ , $\mathbf{z}$ , and $\mathbf{y}$ represent the instruction, input, and output, respectively.