End-to-end training is a core methodology in deep learning where a complete system is built and tuned jointly to map raw inputs to final outputs. This approach contrasts with traditional machine learning pipelines that assemble individually optimized components, such as a separate, manually engineered feature extractor followed by a shallow classifier. By simultaneously training all layers of computation, the model automatically discovers the optimal representations needed for the task.

End-to-End Training

RNNs can be used to classify entire sequences rather than the tokens within them, known as sequence classification. The final hidden state from the RNN is used as the input to a feedforward network that performs the classification. 
The output from the softmax output from the feedforward classifier together with a cross-entropy loss drives the end-to-end training.

Instead of using the final hidden state, another option is to create a pooling layer, that takes all the hidden layers together as the representation.

University of Michigan - Ann Arbor

To capture the inherently temporal aspects of language, RNN, unlike other feed-forward neural networks, is used in NLP, allowing us to handle sequences of varying lengths.


Applications include:
- Recurrent neural language models: autoregressive generation
- Sequence labeling: part-of-speech (POS) tagging; named entity recognition; structure prediction
- Sequence Classification
- Elman Networks

Application of RNN in NLP

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Named entity recognition is to find and classify spans of text that correspond to names of people, places or organizations. To use sequence labeling for a span-recognition problem, we’ll use a technique called IOB encoding.

Sequence labeling: named entity recognition

The technique (Shannon, 1951) generates a word in a sequence given the previous words, using cross-entropy as the loss function and perplexity as the evaluation method. The sequence ends once it reaches a predefined length. Each softmax layer is to decide the next word based on the current hidden state in the RNN component. The initial input token(s) would depend on the NLP task's nature (e.g. machine translation, question answering, summarization). 

RNN's Approach to Text Generation in NLP (Autoregressive Generation)

In the context of NLP, to assign a label to each word(token) of a sequence, each word was first transformed to its corresponding embedding as the input for the RNN model, each RNN block contains an unrolled simple recurrent network, as well as the weight matrices. After forward inference, the probability distributions for the provided label candidates were generated in a softmax layer, the most likely label was selected in the softmax layer at each step; Cross-entropy is applied as the loss function. In POS tagging. Pre-trained word embeddings serve as inputs.

Learn Before

Related

Learn After