Feature selection is preferred for dimensionality reduction when a dataset contains irrelevant or redundant variables that can simply be discarded. In contrast, feature extraction is used when most of the original information must be retained but explained using fewer, newly constructed dimensions.

Choosing Between Feature Selection and Feature Extraction

Reducing an initial set of raw data down to smaller groups so that they are more managable. Usually used when looking at large data sets with multiple variables. 

University of Michigan - Ann Arbor

Process of reducing the number of random variables under consideration by obtaining a set of principal variables. Dimension reduction methods transform the variables before fitting a least squares model. The two types of dimension reduction are
1. Feature selection, 
which selects a subset of variables from the original dataset, making regression or classification more accurate.
2. Feature extraction, 
which combines variables into features, reducing the amount of data processed while maintaining the accuracy of the original dataset. 

Dimension Reduction

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Excluding irrelevant variables from a multiple regression model. Can be done automatically or manually. 
Important because irrelevant variables decrease the accuracy of your data.

Feature Selection

Feature Extraction

- Principal Component Analysis (PCA)
- Linear Discriminant Analysis (LDA)
- Generalized Discriminant Analysis (GDA)
- Singular Value Decomposition (SVD)

Methods of dimensionality reduction

Dimension reduction methods can be compared based on the type of data structure they preserve:

* **Principal Component Analysis (PCA)**: Focuses on preserving global structure by identifying the axes that account for the largest variance in the data.
* **t-Distributed Stochastic Neighbor Embedding (t-SNE)**: Focuses on preserving local structure (neighborhoods) by mapping high-dimensional data to a lower-dimensional space while maintaining relative distances between nearby points.

Learn Before

Related

Learn After