Learn Before
Critique of an Expert Annotation Plan
A technology firm plans to build a dataset to train a model that can identify subtle logical fallacies and cognitive biases in lengthy, complex legal arguments. Their proposed method is to employ a team of experienced lawyers and logicians to read through thousands of documents and label every instance perfectly. Evaluate the feasibility of this approach, focusing specifically on the challenges related to the data creation process itself.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Evaluation in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Example of Human Annotation Challenge: Long Document Analysis
Critique of an Expert Annotation Plan
A research lab is planning to create a new instruction-tuning dataset using a team of highly skilled human experts. Their goal is to build a model capable of a novel, complex reasoning task. Based on the inherent limitations of manual data generation, which of the following proposed tasks would be the most difficult for the expert annotators to execute consistently and reliably at scale?
Annotation Feasibility for a Legal AI