Question: Discuss why reducing or eliminating regularization techniques such as L2, L1 regularization, or dropout can reduce avoidable bias, and explain why this same action tends to increase variance. Analyze the implications for model tuning.

Sample answer: Regularization techniques like L2, L1, and dropout constrain a model's capacity to fit the training data, which helps prevent overfitting but can also prevent the model from achieving low training error. When these constraints are reduced or removed, the model has more freedom to fit the training data closely, which lowers avoidable bias—the gap between training performance and the desired level of performance. However, this increased flexibility also makes the model more sensitive to the specific noise and quirks of the training data, which increases variance—the gap between training and validation/test performance. This means engineers must weigh whether the current problem is bias-dominated or variance-dominated before deciding to reduce regularization, since doing so trades one type of error for another.

Key points:

• Regularization constrains model capacity/flexibility
• Reducing regularization allows closer fit to training data, lowering avoidable bias
• Reducing regularization increases variance
• This is a trade-off, not a free improvement
• Requires diagnosing whether bias or variance is the dominant problem first

Rubric: Full credit requires explaining how regularization constrains model capacity, why removing it reduces avoidable bias, why it increases variance, and the implication that this is a trade-off requiring diagnosis of the dominant error type before acting.