Although a number of large-scale datasets have been collected for common text classification (TC) tasks in recent years, there remains a need for new datasets for more challenging TC tasks. Examples include question answering with multi-step reasoning, text classification for multilingual documents, and text classification for extremely long documents.

Saint Olaf College

Google

+ New Datasets for More Challenging Tasks
+ Modeling Commonsense Knowledge
+ Interpretable DL Models
+ Memory Efficient Models
+ Few-Shot and Zero-Shot Learning

Challenges and opportunities

Incorporating commonsense knowledge into DL models has the potential to significantly improve model performance, pretty much in the same way that humans leverage commonsense knowledge to perform different tasks. For example, a QA system equipped with a commonsense knowledge base could answer questions about the real world. Commonsense knowledge also helps to solve problems in the case
of incomplete information. Using widely held beliefs about everyday objects or concepts, AI systems can reason based on “default” assumptions about the unknowns in a similar way people do. Although this idea has been investigated for sentiment classification, much more research is required to explore to effectively model and use commonsense knowledge in DL models.

Modeling Commonsense Knowledge

Most DL models are supervised models that require large amounts of domain labels. In practice, it is expensive to collect such labels for each new domain. Fine-tuning a PLM (e.g., BERT and OpenGPT) to a specific task requires much fewer domain labels than training a model from scratch, thus opening opportunities of developing new zero-shot or few-shot learning methods based on PLMs.

Few-Shot and Zero-Shot Learning

While deep learning (DL) models achieve promising performance on challenging benchmarks, they largely lack interpretability. It remains difficult to explain why a model excels on one dataset but underperforms on another, what exactly the model has learned, or what minimal neural network architecture is needed to achieve a certain accuracy. Although mechanisms like attention provide some insight, a detailed theoretical understanding of the underlying behavior and dynamics of these models is still lacking. Resolving this gap is crucial for developing more reliable models curated for various applications, such as text analysis.

Interpretability Challenge in Deep Learning Models

Most modern neural language models require a significant amount of memory for training and inference. To meet the computation and storage constraints of edge applications, these models must be compressed. This is typically achieved by building student models via knowledge distillation or through model compression techniques. Developing a task-agnostic model compression method remains an active research challenge.

Learn Before

Related