The training objective for Stage I in JIT-Agent couples supervised imitation learning with reference-anchored preference optimization into a combined loss $\mathcal{L}_{\text{I}}(\theta) = \mathcal{L}_{\text{I}}^{\text{gen}}(\theta) + \lambda_{\text{pref}} \mathcal{L}_{\text{I}}^{\text{pref}}(\theta)$ with $\lambda_{\text{pref}} \ge 0$.

The supervised fine-tuning loss $\mathcal{L}_{\text{I}}^{\text{gen}}(\theta)$ maximizes the log-likelihood of validated teacher tokens to teach protocol-valid scaffold generation:

$\mathcal{L}_{\text{I}}^{\text{gen}}(\theta) = - \mathbb{E}_{(\tau, \pi_\psi, C_\tau, \mathcal{E}_\tau, h^{\text{teach}}) \sim \mathcal{D}_{\text{I}}} \sum_{j=1}^{|h^{\text{teach}}|} \log p_\theta(y_j^{\text{teach}} \mid y_{<j}^{\text{teach}}, c_\tau)$

The preference loss $\mathcal{L}_{\text{I}}^{\text{pref}}(\theta)$ biases candidate generation toward scaffolds that are simultaneously effective and resource-efficient by anchoring against the frozen Stage-I SFT model $p_{\text{ref}}$, weighted by the value margin $\Delta_{\text{val}}$:

$\mathcal{L}_{\text{I}}^{\text{pref}}(\theta) = - \mathbb{E}_{(\tau, \pi_\psi, C_\tau, \mathcal{E}_\tau, h^+, h^-) \sim \mathcal{D}_{\text{I}}^{\text{pref}}} \left[ \Delta_{\text{val}} \cdot \log \sigma \left( \beta_{\text{pref}} \log \frac{p_\theta(h^+ \mid c_\tau)}{p_\theta(h^- \mid c_\tau)} - \beta_{\text{pref}} \log \frac{p_{\text{ref}}(h^+ \mid c_\tau)}{p_{\text{ref}}(h^- \mid c_\tau)} \right) \right]$

where $\sigma$ is the logistic sigmoid, $\beta_{\text{pref}} > 0$ controls preference sharpness, and $\log p(h \mid \cdot)$ is length-normalized sequence log-likelihood.