logo
How it worksCoursesResearch CommunitiesBenefitsAbout Us
Schedule Demo
Learn Before

  • Fine-tuning LLMs with Labeled Data

Definition of LLM Alignment

LLM alignment refers to the process of guiding a Large Language Model to behave in ways that are consistent with human intentions. The guidance for this process can be derived from various sources that reflect human preferences, such as labeled data and direct human feedback.

0

1

6 months ago

Contributors are:

Gemini AI
Gemini AI
🏆 4

Who are from:

Google
Google
🏆 4

References

Tags

Ch.2 Generative Models - Foundations of Large Language Models

Foundations of Large Language Models

Foundations of Large Language Models Course

Computing Sciences

Related

  • Computational Expense of SFT for Large Language Models

  • Objective of Supervised Fine-Tuning

  • Computational Efficiency of Fine-Tuning Compared to Pre-training

  • Suitability of Fine-Tuning for Aligning with Human Values

  • Definition of LLM Alignment

  • Supervised Fine-Tuning for LLM Alignment

  • A company has a powerful, general-purpose language model that can write essays, answer questions, and summarize articles. They want to adapt this model to perform a new, specialized task: generating concise and helpful summaries of customer support tickets. Which of the following strategies represents the most direct and effective approach to adapt the model's internal parameters for this specific purpose?

  • Designing a Dataset for Model Behavior Adaptation

  • Embedding Task Knowledge into LLM Parameters via Fine-Tuning

  • Supervised Fine-Tuning (SFT) as an Example of Labeled Data Fine-Tuning

  • Diagnosing Unintended Model Behavior After Adaptation

Learn After

  • A research lab has developed a large language model that is highly capable of generating human-like text. However, during testing, they find it frequently produces outputs that are unhelpful, factually inaccurate, or contrary to basic ethical principles. To address this, the lab initiates a new phase of training that specifically uses human preferences and feedback to steer the model's outputs towards being more helpful, honest, and harmless. What is the primary goal of this new training phase?

  • Classification of Instruction Fine-Tuning as an Alignment Problem

  • Evaluating Model Training Objectives

  • Example of Misalignment in Instruction-Following

  • Challenges in Defining Human Preferences for LLM Alignment

  • Analysis of LLM Alignment