A typical n × p data matrix X has min(n − 1, p) distinct principal components, but we are not always interested in all of them. Instead, we should strive for the minimum number of principal components required to understand the data properly. This can be done through examining a scree plot, where we can eyeball the scree plot to choose the minimum number of principal components needed to form a good explanation of the variation in the data.

University of Michigan - Ann Arbor

Principal component analysis (PCA) is a method of unsupervised learning techniques that functions as a method for dimension reduction in regression models.  

As explained in the textbook, the objective of PCA is *to find a low-dimensional representation of the observations that explain a good fraction of the variance* (ISLR).


Principal Components Analysis (PCA)

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

This image does a good job at visualizing what PCA is accomplishing. The first principal component is determined such that it captures most of the variation from the data. The second principal component is then set in a direction that captures thee next most variation in the data. So in this example, we see that the first dimension is representing the data from the bottom left to top right of the graph. Then, the second dimension is placed to explain the variation of the data that isn’t explained by the first principal component.


Visualizing PCA

I found this video really helpful when first learning about PCA!
[www.youtube.com/watch?v=jPmV3j1dAv4](https://www.youtube.com/watch?v=jPmV3j1dAv4) 

Helpful video explaining dimensionality reduction/PCA

Deciding How Many Principal Components to Use

The principal components represent the directions of the data that captures the maximum variance (in other words, most information). More specifically, principal components are the individual dimensions that are found to be "interesting" through PCA.

What are Principal Components?

The concept of interesting is used to find a small number of dimensions, principal components, that captures the most amount of information where interesting is measured by "the amount that the observations vary along each dimension"(James et al., 389).

Concept of Interesting

Proportion of variance is basically how the total variance, sum of variances of all individual principal components, can be explained by each of the individual principal components with respect to the sum. It is the ratio between the variance of one or more particular principal component(s) and the total variance. With multiple principal components, add up their individual variances and divide by the total variance.

The Proportion of Variance Explained

This article uses a small dataset to show how PCA works steps by steps, it has the same result with sklearn library.
- Step 1: Standardize the dataset.
- Step 2: Calculate the covariance matrix for the features in the dataset.
- Step 3: Calculate the eigenvalues and eigenvectors for the covariance matrix.
- Step 4: Sort eigenvalues and their corresponding eigenvectors.
- Step 5: Pick k eigenvalues and form a matrix of eigenvectors.
- Step 6: Transform the original matrix.


Steps Involved in the PCA

Probabilistic PCA is a dimensionality reduction technique that analyzes data via a lower-dimensional latent space. The PCA probability model is a slightly modified factor analysis model that uses $$\textbf{W}\textbf{W}^{T} + \sigma^{2}\textbf{I}$$ as the covariance of $$\textbf{x}$$, where $$\sigma^{2}$$ is a scalar:

$$ \textbf{x} \sim N(\textbf{x}; \textbf{b}, \textbf{W}\textbf{W}^{T} + \sigma^{2}\textbf{I}) $$

which can be equivalently expressed as:

$$ \textbf{x} = \textbf{W}\textbf{h} + \textbf{b} + \sigma \textbf{z} $$

where $$\textbf{z} \sim N(\textbf{z}; 0, \textbf{I})$$ is noise, $$\textbf{x}$$ is a data vector, $$\textbf{h}$$ is a latent variable, and $$\textbf{W}$$ is a matrix of principal axes relating the latent variables to the data.

Probabilistic PCA

Dimension reduction methods can be compared based on the type of data structure they preserve:

* **Principal Component Analysis (PCA)**: Focuses on preserving global structure by identifying the axes that account for the largest variance in the data.
* **t-Distributed Stochastic Neighbor Embedding (t-SNE)**: Focuses on preserving local structure (neighborhoods) by mapping high-dimensional data to a lower-dimensional space while maintaining relative distances between nearby points.

Learn Before

Related