Learn Before
Example of Human Annotation Challenge: Long Document Analysis
An example illustrating the limits of human annotation is the task of analyzing an extremely long document. Experts would find it difficult to exhaustively identify all inconsistencies, subtle biases, or missing key information in a time-efficient manner, showcasing a practical bottleneck in manual data creation.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Computing Sciences
Related
Example of Human Annotation Challenge: Long Document Analysis
Critique of an Expert Annotation Plan
A research lab is planning to create a new instruction-tuning dataset using a team of highly skilled human experts. Their goal is to build a model capable of a novel, complex reasoning task. Based on the inherent limitations of manual data generation, which of the following proposed tasks would be the most difficult for the expert annotators to execute consistently and reliably at scale?
Annotation Feasibility for a Legal AI