Shannon, C. E. (1951). Prediction and entropy of printed English. Bell System Technical Journal, 30, 50–64.

Autoregressive generation reference

Instead of generating a sentence from scratch, the language model can complete a sequence given a specified prefix as following.

1. Pass a specified prefix to the language model using forward inference to produce a sequence of hidden states;
2. Apply autoregressive generation using the hidden state of the last word of the prefix as the starting point of generation;
3. The result of this process is a sequence of words that should be a reasonable completion given the prefix input.

A variation on autoregressive generation

The technique (Shannon, 1951) generates a word in a sequence given the previous words, using cross-entropy as the loss function and perplexity as the evaluation method. The sequence ends once it reaches a predefined length. Each softmax layer is to decide the next word based on the current hidden state in the RNN component. The initial input token(s) would depend on the NLP task's nature (e.g. machine translation, question answering, summarization). 

University of Michigan - Ann Arbor

To capture the inherently temporal aspects of language, RNN, unlike other feed-forward neural networks, is used in NLP, allowing us to handle sequences of varying lengths.


Applications include:
- Recurrent neural language models: autoregressive generation
- Sequence labeling: part-of-speech (POS) tagging; named entity recognition; structure prediction
- Sequence Classification
- Elman Networks

Application of RNN in NLP

The autoregression (AR) method models the next step in the sequence as a linear function of the observations at prior time steps.

The notation for the model involves specifying the order of the model p as a parameter to the AR function, e.g. AR(p). For example, AR(1) is a first-order autoregression model.

The method is suitable for univariate time series without trend and seasonal components.

Below is an example of AR code implementation


Autoregression (AR)

NLG, a subfield of artificial intelligence (AI), is a software process that automatically transforms data into plain-English content.

Natural language generation

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Named entity recognition is to find and classify spans of text that correspond to names of people, places or organizations. To use sequence labeling for a span-recognition problem, we’ll use a technique called IOB encoding.

Sequence labeling: named entity recognition

RNN's Approach to Text Generation in NLP (Autoregressive Generation)

In the context of NLP, to assign a label to each word(token) of a sequence, each word was first transformed to its corresponding embedding as the input for the RNN model, each RNN block contains an unrolled simple recurrent network, as well as the weight matrices. After forward inference, the probability distributions for the provided label candidates were generated in a softmax layer, the most likely label was selected in the softmax layer at each step; Cross-entropy is applied as the loss function. In POS tagging. Pre-trained word embeddings serve as inputs.

RNN's Approach to Sequence Labeling in NLP

RNNs can be used to classify entire sequences rather than the tokens within them, known as sequence classification. The final hidden state from the RNN is used as the input to a feedforward network that performs the classification. 
The output from the softmax output from the feedforward classifier together with a cross-entropy loss drives the end-to-end training.

Instead of using the final hidden state, another option is to create a pooling layer, that takes all the hidden layers together as the representation.

RNN's Approach to Sequence Classification in NLP

Autocorrection is an automatic text validation tool commonly found in your phone's and computer's text editing interfaces. This tool can correct spelling errors and/or contextual errors depending on the algorithms.

Autocorrection

inform code-switching in multilingual contexts

A Survey of Code-switching: Linguistic and Social Perspectives for Language Technologie

Hämäläinen, M., & Alnajjar, K. (2021). Human Evaluation of Creative NLG Systems: An Interdisciplinary Survey on Recent Papers. arXiv preprint arXiv:2108.00308.

Human Evaluation of Creative NLG Systems: An Interdisciplinary Survey on Recent Papers

- To present a comprehensive overview of NLG developments since 2000.
- To provide researchers with synthesis and pointers to relevant research.
- To introduce the field to researchers who are less familiar.

Goals of the NLG survey

- Content determination:  Deciding which information to include in the text under construction.
- Text structuring: Determining in which order information will be presented in the text.
- Sentence aggregation: Deciding which information to present in individual sentences.
- Lexicalisation:  Finding the right words and phrases to express information.
- Referring expression generation:  Selecting the words and phrases to identify domain objects.

NLG tasks.

GRL Book

A survey of embedding models of entities and relationships for knowledge graph completion

In Natural Language Processing, text generation tasks that use existing text as input can be broadly divided into two main categories. The first is text completion, which involves continuing to write based on the provided text. The second is text transformation, which focuses on converting the input text into a new, different text.

Learn Before

Related

Learn After