A team training a reward model observes a peculiar behavior: the model consistently assigns higher scores to generated text that ends with the phrase '...and that is the final answer.', even when the main body of the text is of poor quality. The reward score is calculated by applying a linear transformation to the hidden state vector corresponding to the final token of the input sequence. Which of the following provides the most direct explanation for this behavior?

Critique of the Last-Token Reward Calculation Method

An engineer is implementing a reward model where the final scalar score r is computed from the last hidden state vector h_last using the formula r = h_last * W_r. If the hidden state vector h_last has dimensions of [1 x 4096], what must be the dimensions of the weight matrix W_r for the formula to produce a single scalar value?