Setting baselines for an ad recommendation engine
Case context: You are leading a machine learning team building a system to predict which ads to show to individual users. During planning, your team attempts to establish a human-level performance baseline to help estimate the optimal error rate, but human labelers struggle to consistently pick the right ad.
Question: Based on this scenario, what should you conclude about your ability to estimate the optimal error rate?
Sample answer: Because predicting what ad to show a user is a task that even humans have a hard time solving, you should conclude that it will be very hard to estimate the optimal error rate.
Key points:
- Predicting which ad to show is difficult for humans.
- Human difficulty removes a strong baseline.
- Consequently, estimating the optimal error rate will be hard.
Rubric: The response must recognize that human difficulty in this specific task implies that the optimal error rate will be hard to estimate.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Problems Where Humans Perform Poorly Have Weaker Human-Level Baselines
Which characteristic of a task makes it difficult to estimate the optimal error rate, per Machine Learning Yearning?
Predicting what movie to recommend is cited in Machine Learning Yearning as a task where even humans have a hard time.
For tasks that even humans find hard, estimating the _____ error rate is especially difficult.
Match each concept to its correct description regarding optimal error rate estimation.
Order the steps for assessing whether human performance can guide your optimal error rate target.
Which task does Machine Learning Yearning specifically name as an example where human difficulty makes optimal error rate estimation hard?
Human-level performance is always a reliable proxy for the optimal error rate, regardless of whether humans can do the task well.
Alongside movie recommendation, Machine Learning Yearning cites predicting what _____ to show a user as a task even humans find difficult.
Match each scenario to its implication for optimal error rate estimation.
Order the reasoning steps when working on a task where even humans have a hard time.
Consequences of human difficulty on error rate targets
Setting baselines for an ad recommendation engine
Tasks hindering optimal error rate estimation