Part-of-Speech (POS) tagging is an application of sequence labeling in Natural Language Processing where every word in a sentence is assigned a label corresponding to its part of speech.

Part-of-Speech (POS) Tagging

A common architecture for sequence labeling tasks like Named Entity Recognition (NER) uses BERT. The input sequence is first tokenized, and special tokens like `[CLS]` and `[SEP]` are added to the beginning and end, respectively. This sequence is then passed through the BERT model to obtain a contextualized hidden state representation for each token. Finally, a classification layer is placed on top of each token's hidden state (excluding the special tokens) to predict its corresponding label from a predefined set, such as {B, I, O} for NER.

BERT-based Architecture for Sequence Labeling

Span prediction is an NLP task focused on identifying a continuous segment of text. A prominent example is reading comprehension, where a model is given a query (e.g., $$x_1...x_m$$) and a context passage (e.g., $$y_1...y_n$$) and must locate the span within the context that answers the query. This task can be conceptualized as a form of sequence labeling, where the objective is to predict for each token $$y_j$$ in the context whether it marks the beginning or the end of the desired span.

Span Prediction in NLP

Named Entity Recognition (NER) is a foundational task in Natural Language Processing and a subfield of Information Extraction. It involves a process that automatically detects and classifies key pieces of information, known as named entities, from unstructured text into predefined groups or categories.

Definition of Named Entity Recognition

A model is designed to perform a sequence labeling task by identifying organizations and locations within a text. For each word (token), it must assign one of the following labels: `O` (not an entity), `B-ORG` (beginning of an organization), `I-ORG` (inside an organization), `B-LOC` (beginning of a location), or `I-LOC` (inside a location). Given the sentence 'The United Nations headquarters in New York City is a major landmark', which of the following represents the correct sequence of labels?

Consider two distinct natural language processing tasks. Task 1 aims to categorize an entire document (e.g., a news article) into one of several predefined topics like 'Sports', 'Technology', or 'Politics'. Task 2 aims to identify every mention of a person's name within that same document. Which of these two tasks is fundamentally a sequence labeling problem? Justify your answer by explaining how the structure of the output relates to the structure of the input for the chosen task.

Applicability of Sequence Labeling

A sequence labeling model is trained for a task using a standard tagging scheme where 'B-ENTITY' marks the beginning of a multi-token entity, 'I-ENTITY' marks a token inside the same entity, and 'O' marks a token that is not part of any entity. When processing a sentence, the model outputs the label sequence below for the corresponding tokens. Analyze this output. What is the primary logical error in this predicted label sequence, and what does this error suggest about the model's understanding of how to form entities?

Analyzing a Sequence Labeling Model's Output

A sequence labeling model, such as one used for Named Entity Recognition (NER), outputs a probability distribution over the possible tag set at each position, generating a sequence of distributions $${\mathbf{p}_1, \dots, \mathbf{p}_m}$$. The training or fine-tuning of the model can be performed over these distributions. For example, if $$p_i(\mathrm{tag}_i)$$ represents the model's assigned probability for the correct tag at position $$i$$, the training loss can be defined as the negative likelihood.

Negative Likelihood Loss in Sequence Labeling

Sequence labeling is a machine learning methodology used across many Natural Language Processing (NLP) applications. The core idea is to assign a specific class or tag to each token within a given input sequence. The resulting sequence of labels can then be interpreted to extract linguistic annotations. Common examples of this technique include Part-of-Speech (POS) tagging, where each word is labeled with its grammatical role, and Named Entity Recognition (NER), where tokens are tagged to identify entities like names or locations.

Google

Boston University

https://www.coursera.org/learn/nlp-sequence-models

Sequence Models

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Reference of Foundations of Large Language Models Course

- Represent input words are x<1>


- Output to match is y<1>


- Tx denotes length of X.


- Ty denotes length of y.


Sequence Models: Notation

Each word is represented as a vector, with value 1 for the word and 0 for all other words in the dictionary (similar to one-hot encoding)

Representing  Words

The task of predicting what word/letter comes next, unlike the FNN and CNN, in sequence modeling the current output is dependent on the previous a=input and the length of the input is not fixed.


Why sequence models? 

Given a sequence of words, what is the most probable next word/sentence?

Learn Before

Related

Learn After