Deciding on dev set structure for stock market anomaly detection.
Case context: A machine learning team is developing an algorithm to detect micro-anomalies in high-frequency financial trading data. These anomalies occur at millisecond scales and are impossible for human analysts to identify or evaluate. The team is debating whether to set aside time and resources to manually analyze an Eyeball dev set of misclassified anomalies.
Question: Based on the difficulty of this task for humans, how should the team decide on the inclusion of an Eyeball dev set, and why?
Sample answer: The team should omit the Eyeball dev set. Since human analysts cannot identify or evaluate these anomalies, manually examining misclassified examples would not be helpful because they cannot diagnose why the algorithm made a mistake.
Key points:
- Recommendation to omit the Eyeball dev set.
- Lack of human capability to perform the task well.
- Inability to diagnose why the algorithm made a classification error.
Rubric: The response must recommend omitting the Eyeball dev set and justify it by explaining that because humans cannot do the task well, analyzing errors to determine why the algorithm failed is too difficult.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Why is an Eyeball dev set less helpful when humans cannot perform a task well?
An Eyeball dev set is equally useful regardless of whether humans can perform the task well.
When humans cannot do a task well, it is harder to figure out why the algorithm _____ an example correctly.
Match each scenario to the appropriate Eyeball dev set recommendation from Machine Learning Yearning.
Order the steps a practitioner should follow when deciding whether to include an Eyeball dev set.
A team builds a model to detect ultra-low-frequency seismic signals that no human can reliably identify. What should they do about the Eyeball dev set?
According to Machine Learning Yearning, practitioners might omit the Eyeball dev set when humans cannot perform the task well.
For tasks that humans cannot do well, you might _____ having an Eyeball dev set.
Match each concept to its definition in the context of Eyeball dev set decision-making.
Order the logical reasoning chain that explains why non-human-level tasks reduce the value of the Eyeball dev set.
Evaluate the utility of an Eyeball dev set for non-human-level tasks.
Deciding on dev set structure for stock market anomaly detection.
Explain the primary limitation of manual error analysis on tasks humans cannot perform.