Linear discriminant analysis (LDA) is another means of classification similar to logistic regression, but better suited to handle non-binary classifications (i.e. when there are >=3 possible output classes).  

LDA assumes Gaussian distributions for observations in each class, with means being class specific and a covariance common to all classes. Then with the help of Bayes' theorem, the probability for one observation being in each class can be estimated. The class with the highest probability would be the classification result.

In this assumption, the classification is equivalent to maximize the *discriminant function* 
$$
\delta_{k}(x)=x^{T} \boldsymbol{\Sigma}^{-1} \mu_{k}-\frac{1}{2} \mu_{k}^{T} \boldsymbol{\Sigma}^{-1} \mu_{k}+\log \pi_{k}
$$
where ${\mu}_{k}$ is the mean for observations in the $k$th class, ${\pi}_{k}$ is the prior probability that an observation belongs to the $k$th class, and $\Sigma$ is the common covariance matrix. The *discriminant function* is linear to $x$, which is why the method is called *linear discriminant analysis*.


Linear Discriminant Analysis (LDA)

Principal component analysis (PCA) is a method of unsupervised learning techniques that functions as a method for dimension reduction in regression models.  

As explained in the textbook, the objective of PCA is *to find a low-dimensional representation of the observations that explain a good fraction of the variance* (ISLR).


Principal Components Analysis (PCA)

Similar to Linear discriminant analysis (LDA), a Generalized Discriminant Analysis (GDA) method project the data in the high-dimensional space to a space of fewer dimensions. 
LDA method does not work for nonlinear problems. By mapping the input space into a high dimensional feature space with linear properties, a Generalized Discriminant Analysis (GDA) can deal with nonlinear discriminant analysis.
The mapping process of GDA is similar to the mapping process in Support Vector Machine (SVM).


Generalized discriminant analysis (GDA)

Singular value decomposition (SVD) is an unsupervised dimensionality reduction technique used for feature extraction.  The singular value decomposition of an m x n matrix M is a factorization of the form U$$\sum$$M*.  U is an m x m unitary matrix. $$\sum$$ is a m x n diagonal matrix. V is a v x v unitary matrix.
Image link: [www.wikiwand.com/en/Singular_value_decomposition](https://www.wikiwand.com/en/Singular_value_decomposition)
 

Singular value decomposition (SVD)

- Principal Component Analysis (PCA)
- Linear Discriminant Analysis (LDA)
- Generalized Discriminant Analysis (GDA)
- Singular Value Decomposition (SVD)

University of Michigan - Ann Arbor

Process of reducing the number of random variables under consideration by obtaining a set of principal variables. Dimension reduction methods transform the variables before fitting a least squares model. The two types of dimension reduction are
1. Feature selection, 
which selects a subset of variables from the original dataset, making regression or classification more accurate.
2. Feature extraction, 
which combines variables into features, reducing the amount of data processed while maintaining the accuracy of the original dataset. 

Dimension Reduction

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Pang, Peter. Intuitive Machine Learning. Instagram. Retrieved from [www.instagram.com/machinelearning/](https://www.instagram.com/machinelearning/) 

@machinelearning

Excluding irrelevant variables from a multiple regression model. Can be done automatically or manually. 
Important because irrelevant variables decrease the accuracy of your data.

Feature Selection

Reducing an initial set of raw data down to smaller groups so that they are more managable. Usually used when looking at large data sets with multiple variables. 

Feature Extraction

Dimension reduction methods can be compared based on the type of data structure they preserve:

* **Principal Component Analysis (PCA)**: Focuses on preserving global structure by identifying the axes that account for the largest variance in the data.
* **t-Distributed Stochastic Neighbor Embedding (t-SNE)**: Focuses on preserving local structure (neighborhoods) by mapping high-dimensional data to a lower-dimensional space while maintaining relative distances between nearby points.

Learn Before

Related

Learn After

Learn After

Related