Guidance Sources for LLM Alignment

Desirable Attributes of Aligned LLMs

Aligning Large Language Models with Human Values

Challenges in LLM Alignment

Increased Research in LLM Alignment due to Control Concerns

Instruction Alignment

Necessity of Multiple LLM Alignment Methods

Human Preference Alignment via Reward Models

Inference-Time LLM Alignment

Surge in LLM Alignment Research

Fundamental Approaches to LLM Alignment

Increased Urgency of AI Alignment with Advances in AI Capabilities

Goal of LLM Alignment: Accuracy and Safety

Complexity of Human Values in LLM Alignment

Rapid Pace of Research in LLM Alignment

Post-Pre-training Alignment Steps

A user provides the following input to a large language model: 'My five-year-old has a fever of 103°F. What should I do?'

Response A: 'A fever of 103°F in a five-year-old can be caused by various factors, including viral infections like the flu or bacterial infections like strep throat. Historically, fevers were treated with methods like bloodletting, but today...'

Response B: 'I am not a medical professional. A fever of 103°F in a five-year-old can be serious, and you should contact a doctor or seek emergency medical care immediately for guidance.'

Which response better demonstrates the goal of guiding a model's behavior to be consistent with human intentions, and why?

Analysis of an AI Assistant's Behavior

A large language model, pre-trained on a vast dataset from the internet, is tasked with being a helpful and harmless assistant. When a user asks it to 'write a funny story about a programmer,' the model generates a story that relies on negative and outdated stereotypes for its humor. Which statement best analyzes this situation from the perspective of model alignment?

Example of Alignment: Avoiding Harmful Requests

Reward Models as Human Expert Proxies in LLM Alignment

Pre-train-then-align Method for LLM Development