Fine-Tuning as Maximum Likelihood Estimation

A machine learning engineer is adapting a large, pre-trained language model for a new text classification task. They have a labeled dataset D containing pairs of text inputs (x) and their correct labels (y_gold). The engineer formulates the following objective for the adaptation process, where θ represents the model parameters which are initialized randomly:

$\tilde{\theta} = \arg \min_{\theta} \sum_{(\mathbf{x}, \mathbf{y}_{\text{gold}}) \in \mathcal{D}} \text{Loss}(\mathbf{y}_{\theta}, \mathbf{y}_{\text{gold}})$

What is the primary conceptual error in this formulation for the specific goal of adapting the pre-trained model?

Notation for Parameters in the Fine-Tuning Process

Comparing Optimization Objectives in Model Training

The objective for fine-tuning a pre-trained model is formally expressed as: $(\tilde{\omega}, \tilde{\theta}) = \arg \min_{\omega, \hat{\theta}^{+}} \sum_{(\mathbf{x}, \mathbf{y}_{\text{gold}}) \in \mathcal{D}} \text{Loss}(\mathbf{y}_{\omega, \hat{\theta}^{+}}, \mathbf{y}_{\text{gold}})$ Match each component of this objective function to its correct description.

Application Formula for Fine-Tuned BERT Models