Named entity recognition is to find and classify spans of text that correspond to names of people, places or organizations. To use sequence labeling for a span-recognition problem, we’ll use a technique called IOB encoding.

Sequence labeling: named entity recognition

The technique (Shannon, 1951) generates a word in a sequence given the previous words, using cross-entropy as the loss function and perplexity as the evaluation method. The sequence ends once it reaches a predefined length. Each softmax layer is to decide the next word based on the current hidden state in the RNN component. The initial input token(s) would depend on the NLP task's nature (e.g. machine translation, question answering, summarization). 

RNN's Approach to Text Generation in NLP (Autoregressive Generation)

In the context of NLP, to assign a label to each word(token) of a sequence, each word was first transformed to its corresponding embedding as the input for the RNN model, each RNN block contains an unrolled simple recurrent network, as well as the weight matrices. After forward inference, the probability distributions for the provided label candidates were generated in a softmax layer, the most likely label was selected in the softmax layer at each step; Cross-entropy is applied as the loss function. In POS tagging. Pre-trained word embeddings serve as inputs.

RNN's Approach to Sequence Labeling in NLP

RNNs can be used to classify entire sequences rather than the tokens within them, known as sequence classification. The final hidden state from the RNN is used as the input to a feedforward network that performs the classification. 
The output from the softmax output from the feedforward classifier together with a cross-entropy loss drives the end-to-end training.

Instead of using the final hidden state, another option is to create a pooling layer, that takes all the hidden layers together as the representation.

RNN's Approach to Sequence Classification in NLP

To capture the inherently temporal aspects of language, RNN, unlike other feed-forward neural networks, is used in NLP, allowing us to handle sequences of varying lengths.


Applications include:
- Recurrent neural language models: autoregressive generation
- Sequence labeling: part-of-speech (POS) tagging; named entity recognition; structure prediction
- Sequence Classification
- Elman Networks

University of Michigan - Ann Arbor

University of Texas at San Antonio

RNNs are currently being applied to several applications, including:

 - Temporal analysis—time-series anomaly detection and time-series prediction
 - Computer vision—Image description, video tagging and video analysis
 - NLP—Sentiment analysis, speech recognition, language modeling, machine translation and text generation

Applications of RNN

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Application of RNN in NLP

The RNN-RBM sequence model is a generative model of a sequence of frames $\boldsymbol x^{(t)}$ consisting of an RNN that emits RBM parameters for each time step. This model differs from other approaches because it uses the RNN to emit all of the parametrs of the RBM rather than just varying the bias parameters for each time step. The model is trained by back-propagating the gradient of the loss function through the RNN. The loss function is not applied directly to the RNN outputs, but it is applied to the RBM.

RNN-RBM Sequence Models

Recurrent neural networks in language processing is used as a network that contains a cycle within its network connections so that the value of X input is directly or indirectly dependent on its own earlier outputs as an input.

Learn Before

Related

Learn After