Google

Pre-trained models (PTMs) like BERT are generally well-suited for a variety of downstream NLP tasks. After a model has been fine-tuned for a specific purpose, it can be applied to that task. Some classic examples of such applications include:

- General Evaluation Benchmark

- Question Answering

- Sentiment Analysis

- Named Entity Recognition

- Machine Translation

Applications of PTMs

There is an essential issue for the NLP community that how can we evaluate PTMs in a comparable metric. Thus, large scale benchmark is necessary. 

The General Language Understanding Evaluation (GLUE) benchmark is a collection of nine natural language understanding tasks, including single-sentence classification tasks, pairwise text classification tasks, text similarity task, and relevant ranking task. GLUE benchmark is well-designed for evaluating the robustness as well as generalization of models.

General Evaluation Benchmark

Named Entity Recognition (NER) in information extraction and plays an important role in many NLP downstream tasks. 

In deep learning, most of NER methods are in the sequence labeling framework. The entity information in a sentence will be transformed into the sequence of labels, and one label corresponds to one word. The model is used to predict the label of each word.

Named Entity Recognition

BERT models can be adapted for regression tasks, where the goal is to predict a continuous, real-valued score rather than a discrete class label. This adaptation is achieved by modifying the final prediction network, while keeping the core BERT architecture unchanged from its classification counterpart. For instance, to compute the similarity between two sentences, a Sigmoid layer can be added to the prediction network to ensure the output is a score within a specific range.

Text Regression with BERT Models

As one of the most widely-used applications of BERT, single-text classification processes an input sequence to determine its overall category. The input text is typically formatted as a sequence of tokens, such as `[CLS]` $$x_1 x_2 ... x_m$$. The BERT model receives this sequence and encodes it into a corresponding sequence of vectors. The initial output vector, denoted as $$\mathbf{h}_{\mathrm{cls}}$$ (or $$\mathbf{h}_{0}$$), is typically extracted as the comprehensive representation of the entire input text. A prediction network then takes this single $$\mathbf{h}_{\mathrm{cls}}$$ vector as its input to produce a probability distribution over the possible labels.

Single-Text Classification with BERT Models

Based on the firm's goal described in the case study, which natural language processing task is the most appropriate to apply a pre-trained model to? Justify your answer by explaining how this task directly addresses the firm's specific requirements.

Selecting the Appropriate NLP Task for a Business Need

Match each description of a natural language processing task with the most appropriate application name.

A company uses a fine-tuned pre-trained model to automatically process thousands of customer product reviews. When a review states, 'I am extremely disappointed with this purchase; it stopped working after just one use,' the system assigns it a 'Negative' label. Which primary application of a pre-trained model does this system exemplify?

Learn Before

Related