Google

Feature selection can reduce variance by decreasing the number or type of input features, but it can also increase bias. Small reductions in feature count are unlikely to have a large bias effect, while large reductions are more likely to matter, especially if too many useful features are excluded. In modern deep learning with plentiful data, practitioners are more likely to provide all features and let the algorithm learn which to use, but feature selection can still be useful when the training set is small.

Feature Selection for Variance Reduction

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

What two opposing effects can feature selection have on a model's error components?

Reducing input features from 1,000 to 900 is unlikely to have a large effect on model bias.

In modern deep learning with plentiful data, practitioners are more likely to give _____ features to the algorithm and let it sort out which ones to use.

Match each feature reduction scenario to its likely impact on model bias according to Machine Learning Yearning.

Order the reasoning steps for deciding whether to apply feature selection as a variance-reduction technique.

According to Andrew Ng, under which specific condition is feature selection described as 'very useful'?

In modern deep learning with plentiful data, practitioners have largely shifted away from manual feature selection.

Reducing features from 1,000 to _____ is described as a ~10× reduction that is more likely to have a significant effect on bias.

Match each concept to its correct description in the context of feature selection for variance reduction.

Order the steps for evaluating how much a proposed feature reduction will affect model bias.

Question:
Discuss the trade-offs of using feature selection to reduce variance in a machine learning model. How do the scale of the feature reduction and the size of the training dataset influence the decision to apply this technique?

Sample answer:
Feature selection reduces variance by limiting the number or type of input features, but it risks increasing bias by removing potentially useful information. A small reduction in features typically has a minimal effect on bias, whereas a large reduction (e.g., 10x) significantly increases the risk of higher bias. In modern deep learning with plentiful data, it is generally preferred to feed all features into the algorithm and let it learn which ones matter. However, when the training set is small, feature selection remains a highly useful technique to prevent overfitting and manage variance.

Key points:
- Feature selection reduces variance but might increase bias.
- Small feature reductions are unlikely to cause large bias increases, unlike significant reductions.
- With plentiful data in modern deep learning, practitioners generally provide all features to the algorithm.
- Feature selection is very useful when the training set is small.

Rubric:
Award full credit if the answer identifies the core trade-off (variance reduction vs. bias increase), explains the impact of reduction scale (small vs. large reductions), and correctly applies the context of dataset size (plentiful data favors using all features; small datasets favor feature selection).

Trade-offs of Feature Selection in Deep Learning

Case context:
You are building a diagnostic model with a very limited training dataset of 500 patient records. You initially extract 10,000 features per record. The model suffers from high variance. Your colleague suggests either dropping just 500 features or performing a massive 10x reduction down to 1,000 features to solve the problem.

Question:
As a practitioner, how should you evaluate your colleague's suggestions for feature reduction, and what approach is most justified given the dataset size?

Sample answer:
Given the small training set, applying feature selection is highly justified and very useful for reducing variance. However, reducing the feature count slightly (dropping only 500 out of 10,000) is unlikely to have a huge effect on bias or significantly resolve the variance. A massive 10x reduction to 1,000 features will have a much more significant effect on variance, but you must be careful not to exclude too many useful features, as this could increase bias. Therefore, the significant 10x reduction is the better approach to tackle the high variance, provided useful features are carefully retained.

Key points:
- Feature selection is highly appropriate when the training dataset is small.
- A slight reduction in features is unlikely to impact bias or sufficiently fix variance.
- A significant reduction is more likely to have a substantial effect.
- Care must be taken during significant reductions to avoid discarding useful features and increasing bias.

Rubric:
Award full credit if the response correctly identifies that feature selection is appropriate for a small dataset, notes that a small reduction won't have much effect, and explains that a significant reduction is needed but carries the risk of increasing bias if useful features are lost.

Optimizing Features for a Medical Image Classifier

Question:
How has the availability of plentiful data in modern deep learning changed the way practitioners approach feature selection?

Sample answer:
With plentiful data, modern deep learning practitioners have largely shifted away from manual feature selection. Instead, they are more likely to provide all available features to the algorithm and allow it to sort out which ones to use based on the data.

Key points:
- There has been a shift away from manual feature selection in modern deep learning when data is plentiful.
- Practitioners now typically give all features to the algorithm to let it determine which to use.

Rubric:
The answer must explicitly mention the shift away from feature selection and explain that algorithms are now given all features to sort out automatically.

Learn Before

Related