Learn Before
Annotation Feasibility for a Legal AI
A legal tech startup is developing an AI to analyze lengthy corporate merger agreements and identify all clauses that present a 'significant financial risk'. Their plan is to hire a team of expert corporate lawyers to manually annotate these documents, creating a dataset for model training. What is the primary challenge this startup will face regarding the quality and consistency of their human-generated data, and why does the expertise of the annotators not fully solve this problem?
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Example of Human Annotation Challenge: Long Document Analysis
Critique of an Expert Annotation Plan
A research lab is planning to create a new instruction-tuning dataset using a team of highly skilled human experts. Their goal is to build a model capable of a novel, complex reasoning task. Based on the inherent limitations of manual data generation, which of the following proposed tasks would be the most difficult for the expert annotators to execute consistently and reliably at scale?
Annotation Feasibility for a Legal AI