1Cademy - General Formulation of a Sequence Model

Learn Before

Core Problem Types in NLP Pre-training
General Formula for a Parameterized Function

Formula

General Formulation of a Sequence Model

Diverse NLP problems can be unified under a general sequence model structure, represented by the function $\mathbf{o} = g(x_0, x_1, ..., x_m; \theta)$ . In this formula, $x_0, x_1, ..., x_m$ is the input token sequence, where $x_0$ is a special start-of-sequence symbol (such as $\langle s \rangle$ or $\mathbf{[\mathrm{CLS}]}$ ). The function $g(\cdot; \theta)$ (also written as $g_{\theta}(\cdot)$ ) is a neural network defined by parameters $\theta$ , and $\mathbf{o}$ is the model's output. A common shorthand for the output is $\mathbf{o} = g_{\theta}(x_0, x_1, ..., x_m)$ .