The transcription project began with a choice of performance metrics and desired values for these metrics. The goal was to obtain human-level, 98 percent accuracy. The next step is to rapidly establish a sensible baseline system. The team chose a convolutional network with rectiﬁed linear units, which is good for a vision tasks. Then iteratively reﬁne the baseline and test whether each change makes an improvement.



Set Up the Model

At this point, coverage was still below 90 percent. The team instrument the training and test set performance to determine whether the problem was underﬁtting or overﬁtting. Because training and test set error were so similar, this suggested that the problem was due to either underﬁtting or a problem with the training data. Then, they found input image had been cropped too tightly, which remove some digits of address. So, they expand the width of the crop region, which added 10% to the transcription systems coverage.
Finally, the last few percentage points of the performance came from adjusting hyperparameters.



Adjust the Model

Take the Street View transcription system as an example. The process began with data collection. The cars collected the raw data, and human operators provided labels.


State University of New York at Stony Brook

A good machine learning practitioner needs to know how to choose an algorithm for a particular application and how to monitor and respond to feed back obtained from experiments in order to improve a machine learning system. We recommend the following practical design process:
- Determine your goals—what error metric to use, and your target value forthis error metric. These goals and error metrics should be driven by the problem that the application is intended to solve.
- Establish a working end-to-end pipeline as soon as possible, including the estimation of the appropriate performance metrics.
- Instrument the system well to determine bottlenecks in performance. Diagnose which components are performing worse than expected and whether poor performance is due to overﬁtting, underﬁtting, or a defect in the data or software.
- Repeatedly make incremental changes such as gathering new data, adjusting hyperparameters, or changing algorithms, based on speciﬁc ﬁndings from your instrumentation.

Practical Methodology

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Example: Multi-Digit Number Recognition

-	First, determine whether the performance on the training set is acceptable. If performance on the training set is poor, the learning algorithm is not using the training data that is already available, so there is no reason to gather more data. Instead, try increasing the size of the model by adding more layers or adding more hidden units to each layer.
-	Try improving the learning algorithm (tuning the learning rate hyperparameter). 
-	If large models and carefully tuned optimization algorithms do not work well, then the problem might be the quality of the training data. This suggests starting over, collecting cleaner data, or collecting a richer set of features.
-	If the performance on the training set is acceptable, then measure the performance on a test set. If test set performance is much worse than training set performance, then gathering more data is one of the most eﬀective solutions. 
-	A simple alternative to gathering more data is to reduce the size of the model or improve regularization, by adjusting hyperparameters such as weight decay coeﬃcients, or by adding regularization strategies such as dropout.


Learn Before

Related

Learn After