Google

Once a pre-trained language model's parameters are optimized (denoted as $$\hat{\theta}$$), the decoder model, $$\mathrm{Decoder}_{\hat{\theta}}(\cdot)$$, can be utilized to calculate the probability of a token appearing at any given position within a sequence. Specifically, it computes the conditional probability $$\mathrm{Pr}_{\hat{\theta}}(x_{i+1} | x_0,...,x_i)$$ for the next token $$x_{i+1}$$ based on the preceding context.

Probability Computation with Pre-trained Language Models

Once a model's parameters have been optimized through fine-tuning, the resulting model, denoted as $$F_{\tilde{\omega},\tilde{\theta}}(\cdot)$$, can be used for inference on new, unseen data. For instance, in a text classification task, the new text is first tokenized into a sequence of tokens, represented as $$\mathbf{x}_{\mathrm{new}}$$. This token sequence is then fed into the fine-tuned model, which processes it to generate a probability distribution over the possible classes.

Inference Process with a Fine-Tuned Model

When an encoder model parameterized by $$\theta$$ processes an input sequence $$\mathbf{x}$$ and is followed by a Softmax layer parameterized by a weight matrix $$\mathbf{W}$$, it outputs a sequence of probability distributions. This operation is mathematically expressed as: $$ \begin{bmatrix} \mathbf{p}_1^{\mathbf{W},\theta} \\ \vdots \\ \mathbf{p}_m^{\mathbf{W},\theta} \end{bmatrix} = \mathrm{Softmax}_{\mathbf{W}}(\mathrm{Encoder}_{\theta}(\mathbf{x})) $$ In this formula, each $$\mathbf{p}_i^{\mathbf{W},\theta}$$ represents the conditional output distribution $$\Pr(\cdot|\mathbf{x})$$ at sequence position $$i$$. For notation simplicity, the superscripts $$\mathbf{W}$$ and $$\theta$$ affixed to each probability distribution are sometimes dropped.

Probability Distribution Formula for an Encoder-Softmax Language Model

A language model has been trained on a large corpus of English text. When given the sentence 'The chef carefully seasoned the soup with a pinch of ____.', which of the following best represents the direct output the model calculates for the blank position?

A team is using a language model, pre-trained on a vast corpus of general English text, to evaluate the likelihood of different sentences. They input two grammatically correct sentences into the model to calculate their overall probability. Based on the fundamental principle of how these models compute probabilities, which sentence would be assigned a higher overall probability, and why?

Evaluating Sentence Probability

A language model has been pre-trained exclusively on a large corpus of advanced physics research papers. When given the incomplete sentence, 'The best part of waking up is...', the model assigns a very low probability to the token 'coffee' and a much higher probability to the token 'data'. Explain why the model produces this result, based on how it computes probabilities.

Learn Before

Related