You can use a correlation circle to show the principal components. These visualizations show the relationship between the original datapoint and the new principal components. On the graph below, the vertical line through the center of the circle represents the first principal component, which we can see explains 41.2% of variation in the data. The labeled arrows that are more perpendicular to the first principal component are explained most accurately by the first component. For example, the X400m, X110m hurdle, and X100m are close to each other and close to perpendicular to the first principal component line meaning that these three variable are likely related and can be best explained by the first principal component.



Visualizing Principal Components

http://www.sthda.com/english/articles/31-principal-component-methods-in-r-practical-guide/112-pca-principal-component-analysis-essentials/

Principal Component Methods in R: Practical Guide

The principal components represent the directions of the data that captures the maximum variance (in other words, most information). More specifically, principal components are the individual dimensions that are found to be "interesting" through PCA.

University of Michigan - Ann Arbor

Principal component analysis (PCA) is a method of unsupervised learning techniques that functions as a method for dimension reduction in regression models.  

As explained in the textbook, the objective of PCA is *to find a low-dimensional representation of the observations that explain a good fraction of the variance* (ISLR).


Principal Components Analysis (PCA)

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

This image does a good job at visualizing what PCA is accomplishing. The first principal component is determined such that it captures most of the variation from the data. The second principal component is then set in a direction that captures thee next most variation in the data. So in this example, we see that the first dimension is representing the data from the bottom left to top right of the graph. Then, the second dimension is placed to explain the variation of the data that isn’t explained by the first principal component.


Visualizing PCA

I found this video really helpful when first learning about PCA!
[www.youtube.com/watch?v=jPmV3j1dAv4](https://www.youtube.com/watch?v=jPmV3j1dAv4) 

Helpful video explaining dimensionality reduction/PCA

A typical n × p data matrix X has min(n − 1, p) distinct principal components, but we are not always interested in all of them. Instead, we should strive for the minimum number of principal components required to understand the data properly. This can be done through examining a scree plot, where we can eyeball the scree plot to choose the minimum number of principal components needed to form a good explanation of the variation in the data.

Deciding How Many Principal Components to Use

What are Principal Components?

The concept of interesting is used to find a small number of dimensions, principal components, that captures the most amount of information where interesting is measured by "the amount that the observations vary along each dimension"(James et al., 389).

Concept of Interesting

Proportion of variance is basically how the total variance, sum of variances of all individual principal components, can be explained by each of the individual principal components with respect to the sum. It is the ratio between the variance of one or more particular principal component(s) and the total variance. With multiple principal components, add up their individual variances and divide by the total variance.

The Proportion of Variance Explained

This article uses a small dataset to show how PCA works steps by steps, it has the same result with sklearn library.
- Step 1: Standardize the dataset.
- Step 2: Calculate the covariance matrix for the features in the dataset.
- Step 3: Calculate the eigenvalues and eigenvectors for the covariance matrix.
- Step 4: Sort eigenvalues and their corresponding eigenvectors.
- Step 5: Pick k eigenvalues and form a matrix of eigenvectors.
- Step 6: Transform the original matrix.


Steps Involved in the PCA

Probabilistic PCA is a dimensionality reduction technique that analyzes data via a lower dimensional latent space. The PCA probability model is a slightly modified factor analysis model that uses  $\textbf {{W}}$ $\textbf{{W}}^{ ~T}$ + $\sigma^{2}\textbf{{I}}$ as the covariance of $\textbf{x}$ where  $\sigma^{2}$ is now a scalar: 

$$ \textbf{x}  \sim  {N} (\textbf{x}; \textbf{{b}},\textbf {{W}}\textbf{{W}}^{ ~T} + \sigma^{2}\textbf{{I}}) $$  

which can be equivalently expressed as:

$$ \textbf{x} = \textbf{Wh} +  \textbf{b} + \sigma \textbf{z}      $$

where $\textbf{z}$ $\sim$ $\textbf{N( z ; 0, {I})} $ is noise,  $\textbf{x}$  is a data vector,  $\textbf{h}$  is a latent varibale, and $\textbf{W}$  is a set of principal axes relates the latent variables to the data represented as a matrix. 


Learn Before

Related

Learn After