**MiniLM** is a task-agnostic Transformer compression method introduced by Wang et al. (2020). A compact **student** Transformer is trained to mimic two components of the **teacher's last self-attention layer**: (i) the **scaled dot-product attention distributions** over keys, and (ii) a new **value-relation matrix**, defined as the scaled dot products between value vectors. Distilling only the last layer removes the need to align student-teacher layers explicitly, and an optional **teacher assistant** intermediates very large teacher-student gaps. The resulting student keeps the depth and width chosen by the practitioner (e.g., 6 layers, hidden size 384) while preserving most of the teacher's downstream accuracy on GLUE and SQuAD, and is the backbone family from which the **MiniLM-L6-H384** checkpoint is released.

MiniLM Deep Self-Attention Distillation (Wang et al., 2020)

Model compression was was initially proposed as a knowledge transfer from a large / ensemble “teacher” model into training small “student” models with similar accuracy. This was later known as knowledge distillation. 

University of California, Berkeley

Focus of recent work: 
- (1) Efficient building blocks, e.g. depth wise separate convolution, as seen in MobileNets and ShuffleNets 

- (2) Model Compression and acceleration techniques 

Developing Efficient Deep Models 

Gou, J., Yu, B., Maybank, S.J., & Tao, D. (2021). Knowledge Distillation: A Survey. Int. J. Comput. Vis., 129, 1789-1819.

Link: https://arxiv.org/pdf/2006.05525.pdf

Knowledge Distillation: A Survey

- Parameter pruning and sharing: Focus on removing insignificant parameters from deep neural networks. Sub-categories are model quantization, model binarization, structural matrices, and parameter sharing. 
- Low-rank factorization: Use matrix and tensor decomposition to identify redundant parameters
- Transferred compact convolutional filters: Transfer or compress convolutional filters to remove unnecessary parameters 
- Knowledge Distillation (KD): Filter knowledge from a deep neural network into a smaller network 

Model Compression and Acceleration Method Categories    


Model Compression

Model cascading is an inference-time technique designed to enhance efficiency by strategically using a combination of models with varying capabilities. The process initiates with a fast, less accurate small model handling the input to generate a preliminary output. This result is then evaluated against a set of pre-defined criteria. If the criteria are met, the output is accepted. If not, the input is escalated to a slower, more accurate large model for reprocessing. This hierarchical method substantially lowers computational costs and latency by ensuring the resource-intensive large model is only used for inputs that the small model cannot handle effectively.

Learn Before

Related

Learn After