Google

In Supervised Fine-Tuning, training data is structured as pairs of sequences. The input sequence, denoted as $x = x_0...x_m$, typically consists of a combination of an instruction and user-provided context. The corresponding output sequence, denoted as $y = y_1...y_n$, represents the desired response that the model is trained to generate.

Input and Output Sequences in SFT

The dataset used for the reinforcement learning fine-tuning phase, often denoted as $$\mathcal{D}_{\mathrm{rlft}}$$, is generated dynamically. Each training sample is a pair $$(\mathbf{x}, \mathbf{y}_{\hat{\theta}^+})$$. The input sequence $$\mathbf{x}$$ is drawn from a pre-compiled dataset of inputs. The output $$\mathbf{y}_{\hat{\theta}^+}$$, however, is not a fixed pre-existing label; rather, it is sampled from the probability distribution $$\mathrm{\Pr}_{\hat{\theta}^+}(\mathbf{y}|\mathbf{x})$$ defined by the current policy of the language model, which is initialized with pre-trained parameters $$\hat{\theta}$$ and iteratively fine-tuned to reach optimal parameters $$\tilde{\theta}$$.

Dataset Composition for RL Fine-Tuning in RLHF

A machine learning engineer is creating a dataset to fine-tune a language model to act as a helpful assistant. The goal is to teach the model to follow instructions and provide complete, high-quality answers. Which of the following examples represents the most effective input-output pair for this supervised fine-tuning task?

During fine-tuning, each data sample from the tuning dataset (denoted as $$\mathcal{D}_{\mathrm{tune}}$$) is conceptually divided into two distinct components: an input segment, represented as $$\mathbf{x}_{\mathrm{sample}}$$, and an output segment, represented as $$\mathbf{y}_{\mathrm{sample}}$$. These segments are then structured as a single sequence for model processing, typically by concatenating them, which is formally expressed as: $$\mathrm{sample} = [\mathbf{y}_{\mathrm{sample}}, \mathbf{x}_{\mathrm{sample}}]$$.

Structuring a Sample from Input and Output Segments

Analyze the following data sample prepared for Supervised Fine-Tuning. Identify the complete 'input sequence' and the 'output sequence' that the model is being trained on. Explain your reasoning for this division.

Deconstructing an SFT Training Sample

A developer is creating a dataset to fine-tune a language model for text summarization. Below is a source text and the desired summary. Your task is to define the 'input sequence' and the 'output sequence' that would constitute a single training example for this Supervised Fine-Tuning task.

**Source Text:**
"Quantum computing is an emerging field that harnesses the principles of quantum mechanics to solve problems too complex for classical computers. Unlike classical bits, which can be a 0 or a 1, quantum bits or 'qubits' can exist in a superposition of both states simultaneously. This property, along with entanglement, allows quantum computers to perform a vast number of calculations at once."

**Desired Summary:**
"Quantum computing uses quantum mechanics principles like superposition and entanglement, allowing its 'qubits' to process vast calculations simultaneously, tackling problems beyond the reach of classical computers."

Based on the above, what would be the input sequence and the output sequence?

Learn Before

Related