Emergence of Factual Knowledge from Self-Supervision
A large language model is trained on a massive dataset of internet text with the sole objective of predicting a randomly masked word in a sentence. Explain how this simple task enables the model to learn factual information, such as 'The capital of France is Paris,' even though this fact is never explicitly labeled as 'fact' in the training data.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
An AI system is developed by training it on a vast digital library containing only fictional novels written in the 1800s. The system's sole training objective is to repeatedly predict missing words within sentences from these books. If this system is later asked, 'What is the primary method for long-distance communication today?', which statement best evaluates the most likely and significant weakness in its response?
AI Training Strategy for Customer Support
Emergence of Factual Knowledge from Self-Supervision