Evaluating Zero-Shot Generalization in a High-Stakes Application
A hospital plans to deploy a large language model to summarize patient-doctor conversations into medical notes. The model has been pre-trained on a vast corpus of general internet text, giving it a strong ability to understand and follow instructions it has never seen before. However, it has not been specifically trained on any medical data. Evaluate the proposal to use this model for the task 'as-is', relying solely on its pre-existing ability to generalize from instructions. In your evaluation, first explain the core principle that allows the model to attempt this task without specific training, and then argue for or against the proposal, identifying potential risks and benefits.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Evaluation in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A large language model, trained on a vast and diverse corpus of internet text, is given the following novel instruction: 'Analyze the sentiment of the following customer review and express it as a single emoji.' The model has never been explicitly trained on this specific 'text-to-emoji' task. Despite this, it correctly outputs a '😠' for a negative review. Which statement best explains the model's ability to successfully complete this new task?
Evaluating a Development Strategy for a New AI Feature
Evaluating Zero-Shot Generalization in a High-Stakes Application