Learn Before
Distilling Prompting Knowledge into Soft Prompts
The principles of knowledge distillation can be broadly applied to transfer prompting knowledge into a student model's parameters. By training the student model to replicate the outputs of a teacher model, the knowledge embedded in the prompt is effectively distilled. Consequently, the student model can be viewed as having encoded this distilled knowledge in the form of a soft prompt.
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Components of a Knowledge Distillation System
Extensions
Applications
KD Workflow
Distilling Prompting Knowledge into Soft Prompts
Efficient Model Deployment for Mobile Applications
A machine learning team is developing a compact model for a mobile application. They have a large, highly accurate 'teacher' model and a smaller 'student' model architecture. Instead of training the student model directly on the original dataset with its ground-truth labels (e.g., 'this image is a cat'), they train it to mimic the full output probability distribution of the teacher model (e.g., '90% cat, 5% dog, 1% tiger...'). Why is this technique often more effective for the student model's performance than training it from scratch on the original labels?
Mechanisms of Knowledge Transfer
Context Distillation
Learn After
Applications of Prompt Distillation
Optimizing a Language Model for Mobile Deployment
A team aims to create a smaller, more efficient language model that can perform a specific, complex task without requiring the original, lengthy instruction prompt. They decide to transfer the knowledge from the prompt into the model's parameters. Arrange the steps of this process in the correct logical order.
Analyzing the Prompt Distillation Process