TGM is a language model, specifically a neural language model estimating the probability distribution of a token or word.

TGM estimates parameters by minimising the following objective function

$\mathcal L(p_{\theta},\mathcal D) = - \sum_{j=1}^{|\mathcal{D}|} \sum_{t=1}^{|\textbf {x}^{(j)}|} log p_{\theta}( x_t^{(j)} \mid x_1^{(j)},...,x_i^{(j)},....,x_{(t-1)}^{(j)})$

$\textbf{x}_i \in \mathcal{V}$ - Vocabulary of words

textbf{x} = (x_1,....,x_{|textbf{x} |}) - text sequence

$p_*(\textbf{x})$ - reference distribution

$\mathcal D$ - a finite set of text sequences from $p_*$

$p_{\theta}( x_t \mid x_1,...,x_i,....,x_{(t-1)})$ - Probability of next token given the previous tokens in a given sentence