 - Meta information about the article such as author, date of creation, source domain will be appended as extra token in addition to the input sequence and sent for training 

 - Meta information allows the TGM to learn the relationship between original content and the meta information.

 - First controllable TGM proposed is the GROVER model which can create trustworthy fake news content using meta-information of the news article (such as headline, author, and date).  that is harder for humans to identify than human written fake news and can thus pose a significant threat.


Training with control tokens for Controllable Text Generative Models

 In this type of text generation a pre-trained TGM model is combined with one or more attribute classifiers which govern text generation.

Control using attribute classifier for Controllable Text Generative Models

It is possible to control the topic and sentiment of text generated ,two methods by which we can control the text generated are

 - Training with control tokens
 - Control using attribute classifier


San Jose State University

Machine learning models that can generate text  have attained human level of perfection in the areas of grammar, world knowledge, coherence and fluency.

Automatic Detection of Machine Generated Text

Automatic Detection of Machine Generated Text: A Critical Survey
by Ganesh Jawahar, Muhammad Abdul-Mageed, Laks V.S. Lakshmanan
https://arxiv.org/pdf/2011.01314.pdf

Automatic Detection of Machine Generated Text: A Critical Survey


 - Story generation 
 - Conversational response generation
 - Code auto-completion
 - Radiology report generation


Uses of Text Generative Models(TGMs)

 - Fake news generation 
 - Fake product reviews generation 
 - Spamming and phishing


Misuse of Text Generative Models(TGMs)

 - accurate
 - data-efficient
 - generalizable
 - interpretable
 - robust

Qualities of an ideal TGM detector

 - Bias
 - Beneficial usage
 - Malicious usage


Social impacts of TGMs

Most of the latest TGM’s used Transformer model architecture ,it helps to learn long range dependency information and the generated texts are correct ,coherent and uses world knowledge.

Text Generative Model Architecture

Controllable Text Generative Models

Detectors are used to identify machine generated text from human generated text. Classification of detectors are based on

- Classifiers trained from scratch 
 - Zero-shot classifier 
 - Fine-tuning NLM 
 - Human-machine collaboration


TGM Classifiers or Detectors

A system that leverages the use of human visual skills and common sense.

- Differences in human and machine detector
 - Supporting untrained humans
 - Real or Fake Text (RoFT) tool     


Human-machine collaboration for TGMs

A Text Generative Model (TGM) is a neural language model that estimates the probability distribution of tokens. It learns parameters $$\theta$$ by minimizing the following negative log-likelihood objective function over a dataset $$\mathcal{D}$$:

$$ \mathcal{L}(p_{\theta}, \mathcal{D}) = - \sum_{j=1}^{|\mathcal{D}|} \sum_{t=1}^{|\textbf{x}^{(j)}|} \log p_{\theta}(x_t^{(j)} \mid x_1^{(j)}, \dots, x_{t-1}^{(j)}) $$

Where:
- $$x_i \in \mathcal{V}$$ is a token from the vocabulary of words.
- $$\textbf{x} = (x_1, \dots, x_{|\textbf{x}|})$$ is a text sequence.
- $$p_*(\textbf{x})$$ is the reference distribution.
- $$\mathcal{D}$$ is a finite set of text sequences drawn from $$p_*$$.
- $$p_{\theta}(x_t \mid x_1, \dots, x_{t-1})$$ is the model's predicted probability of the next token given the previous tokens in the sequence.

Learn Before

Related

Learn After