Instead of just using small models to generate synthetic data, one can incorporate knowledge distillation loss based on these models. The knowledge distillation loss, denoted as $$\mathrm{Loss}_{\mathrm{kd}}$$, quantifies the difference between the output probability distributions of a teacher (or small) model and a student (or large) model. It is formally defined using the Kullback-Leibler (KL) divergence as:

$$\mathrm{Loss}_{\mathrm{kd}} = \mathrm{KL}(\mathrm{Pr}^{w}(\cdot|\mathbf{x})\ ||\ \mathrm{Pr}^{s}_{\theta}(\cdot|\mathbf{x}))$$

Here, $$\mathrm{Pr}^{w}(\cdot|\mathbf{x})$$ is the probability distribution produced by the teacher (or weak) model, and $$\mathrm{Pr}^{s}_{\theta}(\cdot|\mathbf{x})$$ is the distribution from the student model with parameters $$\theta$$, given an input $$\mathbf{x}$$. This simple loss function measures the difference between the small and large models and is minimized to encourage the large model to mimic the small model's behavior.

Knowledge Distillation Loss using KL Divergence

Most modern neural language models require a significant amount of memory for training and inference. To meet the computation and storage constraints of edge applications, these models must be compressed. This is typically achieved by building student models via knowledge distillation or through model compression techniques. Developing a task-agnostic model compression method remains an active research challenge.

Saint Olaf College

Google

+ New Datasets for More Challenging Tasks
+ Modeling Commonsense Knowledge
+ Interpretable DL Models
+ Memory Efficient Models
+ Few-Shot and Zero-Shot Learning

Challenges and opportunities

Incorporating commonsense knowledge into DL models has the potential to significantly improve model performance, pretty much in the same way that humans leverage commonsense knowledge to perform different tasks. For example, a QA system equipped with a commonsense knowledge base could answer questions about the real world. Commonsense knowledge also helps to solve problems in the case
of incomplete information. Using widely held beliefs about everyday objects or concepts, AI systems can reason based on “default” assumptions about the unknowns in a similar way people do. Although this idea has been investigated for sentiment classification, much more research is required to explore to effectively model and use commonsense knowledge in DL models.

Modeling Commonsense Knowledge

Most DL models are supervised models that require large amounts of domain labels. In practice, it is expensive to collect such labels for each new domain. Fine-tuning a PLM (e.g., BERT and OpenGPT) to a specific task requires much fewer domain labels than training a model from scratch, thus opening opportunities of developing new zero-shot or few-shot learning methods based on PLMs.

Few-Shot and Zero-Shot Learning

While deep learning (DL) models achieve promising performance on challenging benchmarks, they largely lack interpretability. It remains difficult to explain why a model excels on one dataset but underperforms on another, what exactly the model has learned, or what minimal neural network architecture is needed to achieve a certain accuracy. Although mechanisms like attention provide some insight, a detailed theoretical understanding of the underlying behavior and dynamics of these models is still lacking. Resolving this gap is crucial for developing more reliable models curated for various applications, such as text analysis.

Interpretability Challenge in Deep Learning Models

Memory Efficiency Challenge in Neural Language Models

Although a number of large-scale datasets have been collected for common text classification (TC) tasks in recent years, there remains a need for new datasets for more challenging TC tasks. Examples include question answering with multi-step reasoning, text classification for multilingual documents, and text classification for extremely long documents.

Learn Before

Related

Learn After