When preprocessing a dataset for machine learning, features that serve solely as unique identifiers (such as ID numbers) must be removed before feeding the data into the model. While these identifiers are convenient for distinguishing specific training examples, they carry no relevant information for prediction purposes and can introduce noise or spurious correlations into the learning process.

Identifier Feature Removal

Excluding irrelevant variables from a multiple regression model. Can be done automatically or manually. 
Important because irrelevant variables decrease the accuracy of your data.

University of Michigan - Ann Arbor

Process of reducing the number of random variables under consideration by obtaining a set of principal variables. Dimension reduction methods transform the variables before fitting a least squares model. The two types of dimension reduction are
1. Feature selection, 
which selects a subset of variables from the original dataset, making regression or classification more accurate.
2. Feature extraction, 
which combines variables into features, reducing the amount of data processed while maintaining the accuracy of the original dataset. 

Dimension Reduction

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Feature Selection

Reducing an initial set of raw data down to smaller groups so that they are more managable. Usually used when looking at large data sets with multiple variables. 

Feature Extraction

Feature selection: Use feature selection as a form of dimensionality reduction when your data has dimensions that are not relevant to the problem you are trying to solve or if you have repeating information within dimensions. In these cases, simply remove the unnecessary dimensions.

Feature extraction: Use feature extraction for dimensionality reduction when you want to use most of the information from your data set, but need the information explained in fewer dimensions. Also, feature extraction will take care of some of the repeat information from your data. 

When to use feature selection vs. feature extraction

- Principal Component Analysis (PCA)
- Linear Discriminant Analysis (LDA)
- Generalized Discriminant Analysis (GDA)
- Singular Value Decomposition (SVD)

Methods of dimensionality reduction

PCA 
-identifies the axis that accounts for the largest amount of variance
- tries to preserve global structure of data 
t-SNE
- A powerful manifold learning method that finds a 2D projection preserving information about neighbors
- Gives much more weight to preserving information about distances between points that are neighbors
- Work better on datasets that have more well defined local structure


Learn Before

Related

Learn After