A reward model is constructed by taking a large, pre-trained language model and adding a new linear layer on top to output a single scalar value. To train this model efficiently, an engineer freezes the weights of the pre-trained language model and only updates the weights of the new linear layer. How does this training strategy relate to the complete set of the reward model's parameters, denoted as ϕ?

Components of Reward Model Parameters

In the context of training a reward model, the parameter set ϕ, which is optimized to minimize the loss function, consists solely of the weights of the final linear layer responsible for mapping the model's internal representations to a scalar reward score.