Learn Before
Components of a Knowledge Distillation System
- Knowledge
- Distillation algorithm
- Teacher-student architecture
0
1
References
Tags
Deep Learning (in Machine learning)
Data Science
Related
Extensions
Applications
KD Workflow
Distilling Prompting Knowledge into Soft Prompts
Efficient Model Deployment for Mobile Applications
A machine learning team is developing a compact model for a mobile application. They have a large, highly accurate 'teacher' model and a smaller 'student' model architecture. Instead of training the student model directly on the original dataset with its ground-truth labels (e.g., 'this image is a cat'), they train it to mimic the full output probability distribution of the teacher model (e.g., '90% cat, 5% dog, 1% tiger...'). Why is this technique often more effective for the student model's pe
Mechanisms of Knowledge Transfer
Context Distillation
Learn After
Related
Extensions
Applications
KD Workflow
Distilling Prompting Knowledge into Soft Prompts
Efficient Model Deployment for Mobile Applications
A machine learning team is developing a compact model for a mobile application. They have a large, highly accurate 'teacher' model and a smaller 'student' model architecture. Instead of training the student model directly on the original dataset with its ground-truth labels (e.g., 'this image is a cat'), they train it to mimic the full output probability distribution of the teacher model (e.g., '90% cat, 5% dog, 1% tiger...'). Why is this technique often more effective for the student model's pe
Mechanisms of Knowledge Transfer
Context Distillation