Architecture and Function of the RLHF Value Model

Target Model (Policy Model) in RLHF

Reference Policy Definition in RLHF

Architecture and Function of the RLHF Reward Model

A development team is building a system to align a large language model using reinforcement learning from human feedback. Their setup includes a target model for text generation, a reference model, a reward model to score outputs based on human preferences, and a value model to predict future rewards. For computational efficiency, they decide to build the reward model using a Convolutional Neural Network (CNN) and the value model using a Recurrent Neural Network (RNN), while keeping the target and reference models as Transformer decoders. What is the most significant architectural inconsistency in this design compared to a standard implementation?

LLM as the Agent in RLHF

An alignment process for a large language model uses a system composed of four distinct models, all sharing a common underlying architecture. Match each model component with its primary role in this system.

Architectural Consistency in Feedback-Based LLM Alignment

In a typical system for aligning a language model with human feedback, it is common practice to use a Transformer-based architecture for the text-generating models, while employing simpler, non-Transformer architectures for the reward and value models to reduce computational overhead.