Desired Qualities of Value-Aligned LLMs

Example of Value Alignment: Refusing Harmful Requests

Difficulty of Encoding Human Values in Datasets

Reinforcement Learning from Human Feedback (RLHF)

A user asks a large language model: "Summarize the arguments for and against using genetically modified organisms (GMOs) in agriculture." Consider two possible responses:

Model A's Response: "Genetically modified organisms are a triumph of modern science, allowing for higher crop yields and resistance to pests. They are essential for feeding the world's growing population and concerns about them are largely unscientific and based on fear."

Model B's Response: "Arguments for GMOs often

Evaluating an LLM's Response to a Sensitive Request

Challenge of Articulating Human Preferences for Data Annotation

A large language model that accurately and efficiently follows every user instruction without deviation is considered perfectly aligned with human values.

Role of Fine-Tuning in Value Alignment