Learn Before
Applications
Student models can be applied to visual and speech recognition and NLP. Knowledge distillation has been used for label smoothing, accessing the teacher’s accuracy, and for obtaining a prior for the optimal output layer geometry.
0
1
Tags
Deep Learning (in Machine learning)
Data Science
Related
Components of a Knowledge Distillation System
Extensions
Applications
KD Workflow
Distilling Prompting Knowledge into Soft Prompts
Efficient Model Deployment for Mobile Applications
A machine learning team is developing a compact model for a mobile application. They have a large, highly accurate 'teacher' model and a smaller 'student' model architecture. Instead of training the student model directly on the original dataset with its ground-truth labels (e.g., 'this image is a cat'), they train it to mimic the full output probability distribution of the teacher model (e.g., '90% cat, 5% dog, 1% tiger...'). Why is this technique often more effective for the student model's performance than training it from scratch on the original labels?
Mechanisms of Knowledge Transfer
Context Distillation