Shift in LLM Alignment from Predefined Tasks to Real-World Interaction

Impracticality of Achieving Alignment Solely Through Pre-training

Need for Diverse Alignment Methods

Insufficiency of Data Fitting for Value Alignment

Difficulty of Encoding Human Values in Datasets

Inarticulacy of Human Preferences as an Alignment Challenge

Goodhart's Law

Real-World Complexity as an Alignment Challenge

Specification Gaming in AI Alignment

Alignment Challenges as a Motivator for AI Research

Diversity and Fluidity of Human Values as an Alignment Challenge

Analysis of an LLM Alignment Failure

A development team building a chatbot aims for it to be 'helpful' to all users. They discover that behaviors praised as helpful by users in one country are criticized as intrusive by users in another. This issue persists even after training the model on vast, culturally diverse datasets. Which fundamental challenge in guiding a model's behavior does this scenario best illustrate?

Evaluating Core Difficulties in Model Behavior Guidance

Challenge of Defining Human Values for AI Objectives