Learn Before
Concept
Limitations of Pre-fixed Teacher-Student Architectures in Knowledge Distillation
In traditional knowledge distillation, the teacher and student model architectures are almost always pre-fixed in size and structure. This fixed configuration often results in a model capacity gap, where the smaller student cannot effectively absorb the larger teacher's knowledge. Furthermore, standard distillation frameworks lack a systematic methodology to design these architectures and do not explain how the pre-fixed model setups constrain the architectural choices.
0
1
Updated 2026-07-06
Contributors are:
Who are from:
Tags
Deep Learning (in Machine learning)
Data Science
Computing Sciences