Question: Define the blackbox dev set, name three ways it can help during model development, and explain the rule for how people should interact with it.

Sample answer: A blackbox dev set is the part of the development data reserved for automated checking after an eyeball dev set has been separated out. It is useful for computing model error rates, comparing alternative methods, and tuning hyperparameters. The key rule is that developers should not inspect this set manually; it should be used only by automated evaluation so that the results stay unbiased.

Key points:

• It is the remaining portion of the dev data after the eyeball dev set is set aside.
• It supports automated error measurement, method comparison, and hyperparameter tuning.
• It should not be manually examined by developers.

Rubric: The response must state that the blackbox dev set is the leftover dev subset after the eyeball dev set is removed. It must identify all three uses: automatic evaluation of model performance, comparison among candidate approaches, and hyperparameter tuning. It must also explain that developers should not look through the examples themselves, so the set remains suitable for objective automated assessment.