In a reinforcement learning process for training a language model, a 'value model' is used to estimate the expected future reward from any given point in a generated text sequence. What is the primary analytical reason for updating this value model's parameters after each token is generated, rather than only once at the end of the complete sequence?

Diagnosing Inefficient Language Model Training

During the iterative process of training a language model using human feedback, the component responsible for estimating future rewards (the 'value model') is only updated once, after an entire sequence of text has been fully generated.