For one class of entity, we label any token that begins a span of named entity with the label B, tokens that occur inside a span are tagged with an I, and any tokens outside of any span of interest are labeled O.

IOB encoding (single entity)

- Choosing the maximum probability label for each element in a sequence independently does not necessarily result in an optimal (or even very good) sequence of tags.
- It doesn't guarantee the resulting sequence is well-formed or legal. For example, tag O might be followed by tag I; a tag I-LOC following a tag B-PER.

Drawbacks of IOB tagging

Many NLP systems make use of NER systems:
- Search engine efficiency
- Recommendation engine
- Customer service (assigning proper agent)
- Automatic trading

Applications of Named Entity Recognition (NER) 

The standard IOB (Inside, Outside, Beginning) tagging scheme uses the tag set $${\mathrm{B}, \mathrm{I}, \mathrm{O}}$$ to identify entities. When extending this to multiple entity types, the 'B' and 'I' tags are specialized for each class. For example, $$\mathrm{B}\textrm{-}\mathrm{ORG}$$ signifies the beginning of an organization's name, $$\mathrm{I}\textrm{-}\mathrm{ORG}$$ indicates a word inside an organization, and $$\mathrm{O}$$ means the word does not belong to any named entity. This specialization expands the total tag set size to $${}2 \times N + 1$$, where $$N$$ is the number of distinct entity classes.

IOB Encoding for Multiple Entity Types

Named entity recognition is to find and classify spans of text that correspond to names of people, places or organizations. To use sequence labeling for a span-recognition problem, we’ll use a technique called IOB encoding.

University of Michigan - Ann Arbor

To capture the inherently temporal aspects of language, RNN, unlike other feed-forward neural networks, is used in NLP, allowing us to handle sequences of varying lengths.


Applications include:
- Recurrent neural language models: autoregressive generation
- Sequence labeling: part-of-speech (POS) tagging; named entity recognition; structure prediction
- Sequence Classification
- Elman Networks

Application of RNN in NLP

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Sequence labeling: named entity recognition

The technique (Shannon, 1951) generates a word in a sequence given the previous words, using cross-entropy as the loss function and perplexity as the evaluation method. The sequence ends once it reaches a predefined length. Each softmax layer is to decide the next word based on the current hidden state in the RNN component. The initial input token(s) would depend on the NLP task's nature (e.g. machine translation, question answering, summarization). 

RNN's Approach to Text Generation in NLP (Autoregressive Generation)

In the context of NLP, to assign a label to each word(token) of a sequence, each word was first transformed to its corresponding embedding as the input for the RNN model, each RNN block contains an unrolled simple recurrent network, as well as the weight matrices. After forward inference, the probability distributions for the provided label candidates were generated in a softmax layer, the most likely label was selected in the softmax layer at each step; Cross-entropy is applied as the loss function. In POS tagging. Pre-trained word embeddings serve as inputs.

RNN's Approach to Sequence Labeling in NLP

RNNs can be used to classify entire sequences rather than the tokens within them, known as sequence classification. The final hidden state from the RNN is used as the input to a feedforward network that performs the classification. 
The output from the softmax output from the feedforward classifier together with a cross-entropy loss drives the end-to-end training.

Instead of using the final hidden state, another option is to create a pooling layer, that takes all the hidden layers together as the representation.

Learn Before

Related

Learn After