If the dev set is not big enough to split, the entire dev set can be used as the Eyeball dev set for manual error analysis, model selection, and hyperparameter tuning.

Google

When the dev set is large, for example 5,000 examples with a 20% error rate yielding about 1,000 misclassified images, manually examining all misclassified examples takes a long time, so one may decide not to use all of them for error analysis. In that case, the dev set can be explicitly split into two subsets: one that is manually looked at and one that is not. The portion that is manually examined will be overfit more rapidly, while the portion that is not manually looked at can be used to tune parameters. Splitting the dev set in this explicit way also lets one tell when the manual error analysis process is causing overfitting of the examined portion.

Splitting a Large Dev Set into a Manually Examined Subset and a Hands-Off Subset

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

The Eyeball dev set is the portion of a split dev set that one manually examines for error analysis. It is named Eyeball as a reminder that it is being looked at with the eyes; for a speech recognition project in which one listens to audio clips, it might instead be called an Ear dev set. It is created by randomly selecting some fraction of the dev set, for example 10%, with the aim of including roughly 100 misclassified examples to look at. Continuing the running example, 10% of a 5,000-example dev set produces an Eyeball dev set of 500 examples, of which the algorithm is expected to misclassify about 100.

Eyeball Dev Set

The Blackbox dev set is the second subset of a split dev set, consisting of the remaining examples after the Eyeball dev set is drawn; in the running example it contains 4,500 examples. It can be used to evaluate classifiers automatically by measuring their error rates, to select among algorithms, and to tune hyperparameters. One should avoid looking at it with one's eyes. The term Blackbox is used because this subset is only used to obtain blackbox evaluations of classifiers.

Learn Before

Related