University of Michigan - Ann Arbor

A Basic Supervised Statistical Learning Workflow

Which one shows a basic machine learning workflow?

In this step, we choose:
- A feature representation of the dataset, e.g., image pixels
- Type of classifier to use, e.g., K-nearest neighbor classifier

Classification Problem Representation

In this step, we decide on the criterion to distinguish good vs. bad classifiers. For example, the percentage of correct predictions on test set.
The learned models will be evaluated on evaluation functions (also called objective functions or scoring functions) in order to distinguish between successful and unsuccessful models. The desired, external evaluation function might be different from the one the algorithm uses.

Classification Problem Evaluation

In this step, we search for the most optimal values for (hyper)parameters that give the best classification model for the chosen evaluation criterion. For example, we can try a range of values for the "k" parameter in k-nearest neighbor classifier.

Classification Problem Optimization

For a supervised learning algorithm to do well, you need to check the following four assumptions:
- **Low Bias**: Fits training set well on cost function (e.g., comparable to human level performance)
- **Low Variance**: Fits dev set on cost function
- Fits test set well on cost function
- Performs well in real world (e.g., happy users)

Chain of Assumptions in Supervised Statistical Learning

Domingos, P. (2012). A few useful things to know about machine learning. Communications of the ACM, 55(10), 78-87. doi:10.1145/2347736.2347755

A few useful things to know about machine learning

•	Data leakage

•	Modeling non-linear data with linear models 

•	Forgetting to regularize data if needed

•	Substandard data cleaning

•	Failing to understand the underlying logic behind certain packages

Learn Before

Related