Learn Before
Automatic Dictation Score Check
Case context: A hospital is developing an automatic dictation system. For one audio clip, the reference transcript S* is "The nurse checked the patient's pulse." The system's output Sout is "The nurse checked the patient's purse." A verification run gives ScoreA(S*) = 0.67 and ScoreA(Sout) = 0.74.
Question: What does this result suggest about the main cause of the error, and which part of the system should you improve first?
Sample answer: The problem is in the scoring objective, because the reference transcript receives a lower score than the incorrect output. The first thing to improve is the component that learns the score for candidate transcripts, rather than the search procedure that selects among them.
Key points:
- Treat this as a scoring/objective issue.
- The reference score is below the wrong-output score.
- Focus on improving the score model.
Rubric: The learner must identify the scoring objective problem and recommend improving the learning component that produces the scores.
0
1
Tags
Data Science
Foundations of Large Language Models Course
Computing Sciences
D2L
Dive into Deep Learning @ D2L
Machine Learning
Deep Learning
Supervised Learning
Machine Learning Yearning @ DeepLearning.AI
Related
Cross-entropy loss
Logistic Regression Cost Function
A machine learning model is being trained for a prediction task. A key metric, the objective function, is tracked over time. The value of this function represents the magnitude of the model's error. A graph of this process shows the objective function's value consistently decreasing as the number of training iterations increases. What is the most accurate interpretation of this trend?
Diagnosing Model Training Issues
Calculating and Interpreting a Model's Objective Function
Surrogate Objective
Loss Function
Differentiable Objectives
Second-Order Optimization Algorithm
Objective Function Curvature
Convex Quadratic Objective Function
Recognizing a Scoring-Function Issue
If a candidate answer gets a lower score than an incorrect answer, the search procedure is usually the part that should be fixed first.
A scoring rule can mislead learning if it gives the correct output a _____ score than the system's prediction.
When to Treat a Scoring Problem as an Objective-Function Issue
When the Scoring Rule Is the Source of the Error
Diagnosing a Scoring-Rule Failure
Automatic Dictation Score Check
Improving the Score Estimator
What an Objective Function Must Do
A Metric Can Still Be Wrong