Value Function Loss in RLHF

An AI system is being trained to generate helpful multi-turn dialogues. A state-value function, which estimates the total future reward from the current point in the conversation, is updated using rewards from a separate reward model. The development team observes that the value function consistently assigns very low values to all conversational turns except the very last one, even when the intermediate turns are crucial for a successful outcome. This causes the AI to prematurely end conversations. Which of the following is the most likely cause of this specific issue?

Impact of a Biased Reward Model on Value Function Training

Advantage Function as TD Error in RLHF

Arrange the following events in the correct chronological order to describe a single update step for a value function that relies on a separate reward model.