Instrumenting a machine learning system involves setting up logging, monitoring, and analytical tools to measure performance across different components of the pipeline. By collecting training and test set metrics, a practitioner can diagnose performance bottlenecks and determine whether system limitations are caused by overfitting, underfitting, or defects in the data or software.

Google

A good machine learning practitioner must know how to select an algorithm for a specific application and how to monitor and respond to experimental feedback to improve the model. In the textbook *Deep Learning*, Ian Goodfellow, Yoshua Bengio, and Aaron Courville recommend a four-step practical design process:

1. **Determine Goals:** Choose appropriate error metrics and set target values that are driven directly by the problem the application is intended to solve.
2. **Establish a Pipeline:** Set up a working, end-to-end pipeline as soon as possible, including tools to estimate the selected performance metrics.
3. **Instrument the System:** Monitor and analyze the pipeline to identify performance bottlenecks. Diagnose whether poor performance is due to overfitting, underfitting, or issues in the software or data.
4. **Iterate and Refine:** Repeatedly make incremental adjustments—such as collecting more data, tuning hyperparameters, or selecting different algorithms—guided by the specific findings from your instrumentation.

Practical Design Process for Machine Learning Systems

Take the Street View transcription system as an example. The process began with data collection. The cars collected the raw data, and human operators provided labels.


Example: Multi-Digit Number Recognition

-	First, determine whether the performance on the training set is acceptable. If performance on the training set is poor, the learning algorithm is not using the training data that is already available, so there is no reason to gather more data. Instead, try increasing the size of the model by adding more layers or adding more hidden units to each layer.
-	Try improving the learning algorithm (tuning the learning rate hyperparameter). 
-	If large models and carefully tuned optimization algorithms do not work well, then the problem might be the quality of the training data. This suggests starting over, collecting cleaner data, or collecting a richer set of features.
-	If the performance on the training set is acceptable, then measure the performance on a test set. If test set performance is much worse than training set performance, then gathering more data is one of the most eﬀective solutions. 
-	A simple alternative to gathering more data is to reduce the size of the model or improve regularization, by adjusting hyperparameters such as weight decay coeﬃcients, or by adding regularization strategies such as dropout.


Learn Before

Related