Learn Before
Concept

Limitations of Pre-fixed Teacher-Student Architectures in Knowledge Distillation

In traditional knowledge distillation, the teacher and student model architectures are almost always pre-fixed in size and structure. This fixed configuration often results in a model capacity gap, where the smaller student cannot effectively absorb the larger teacher's knowledge. Furthermore, standard distillation frameworks lack a systematic methodology to design these architectures and do not explain how the pre-fixed model setups constrain the architectural choices.

0

1

Updated 2026-07-06

Tags

Deep Learning (in Machine learning)

Data Science

Computing Sciences

Learn After