A Text Generative Model (TGM) is a neural language model that estimates the probability distribution of tokens. It learns parameters $\theta$ by minimizing the following negative log-likelihood objective function over a dataset $\mathcal{D}$:

$\mathcal{L}(p_{\theta}, \mathcal{D}) = - \sum_{j=1}^{|\mathcal{D}|} \sum_{t=1}^{|\textbf{x}^{(j)}|} \log p_{\theta}(x_t^{(j)} \mid x_1^{(j)}, \dots, x_{t-1}^{(j)})$

Where:

• $x_i \in \mathcal{V}$ is a token from the vocabulary of words.
• textbf{x} = (x_1, dots, x_{|textbf{x}|}) is a text sequence.
• $p_*(\textbf{x})$ is the reference distribution.
• $\mathcal{D}$ is a finite set of text sequences drawn from $p_*$.
• $p_{\theta}(x_t \mid x_1, \dots, x_{t-1})$ is the model's predicted probability of the next token given the previous tokens in the sequence.