This image does a good job at visualizing what PCA is accomplishing. The first principal component is determined such that it captures most of the variation from the data. The second principal component is then set in a direction that captures thee next most variation in the data. So in this example, we see that the first dimension is representing the data from the bottom left to top right of the graph. Then, the second dimension is placed to explain the variation of the data that isn’t explained by the first principal component.


Visualizing PCA

I found this video really helpful when first learning about PCA!
[www.youtube.com/watch?v=jPmV3j1dAv4](https://www.youtube.com/watch?v=jPmV3j1dAv4) 

Helpful video explaining dimensionality reduction/PCA

A typical n × p data matrix X has min(n − 1, p) distinct principal components, but we are not always interested in all of them. Instead, we should strive for the minimum number of principal components required to understand the data properly. This can be done through examining a scree plot, where we can eyeball the scree plot to choose the minimum number of principal components needed to form a good explanation of the variation in the data.

Deciding How Many Principal Components to Use

The principal components represent the directions of the data that captures the maximum variance (in other words, most information). More specifically, principal components are the individual dimensions that are found to be "interesting" through PCA.

What are Principal Components?

The concept of interesting is used to find a small number of dimensions, principal components, that captures the most amount of information where interesting is measured by "the amount that the observations vary along each dimension"(James et al., 389).

Concept of Interesting

Proportion of variance is basically how the total variance, sum of variances of all individual principal components, can be explained by each of the individual principal components with respect to the sum. It is the ratio between the variance of one or more particular principal component(s) and the total variance. With multiple principal components, add up their individual variances and divide by the total variance.

The Proportion of Variance Explained

This article uses a small dataset to show how PCA works steps by steps, it has the same result with sklearn library.
- Step 1: Standardize the dataset.
- Step 2: Calculate the covariance matrix for the features in the dataset.
- Step 3: Calculate the eigenvalues and eigenvectors for the covariance matrix.
- Step 4: Sort eigenvalues and their corresponding eigenvectors.
- Step 5: Pick k eigenvalues and form a matrix of eigenvectors.
- Step 6: Transform the original matrix.


Steps Involved in the PCA

Probabilistic PCA is a dimensionality reduction technique that analyzes data via a lower-dimensional latent space. The PCA probability model is a slightly modified factor analysis model that uses $$\textbf{W}\textbf{W}^{T} + \sigma^{2}\textbf{I}$$ as the covariance of $$\textbf{x}$$, where $$\sigma^{2}$$ is a scalar:

$$ \textbf{x} \sim N(\textbf{x}; \textbf{b}, \textbf{W}\textbf{W}^{T} + \sigma^{2}\textbf{I}) $$

which can be equivalently expressed as:

$$ \textbf{x} = \textbf{W}\textbf{h} + \textbf{b} + \sigma \textbf{z} $$

where $$\textbf{z} \sim N(\textbf{z}; 0, \textbf{I})$$ is noise, $$\textbf{x}$$ is a data vector, $$\textbf{h}$$ is a latent variable, and $$\textbf{W}$$ is a matrix of principal axes relating the latent variables to the data.

Probabilistic PCA

Dimension reduction methods can be compared based on the type of data structure they preserve:

* **Principal Component Analysis (PCA)**: Focuses on preserving global structure by identifying the axes that account for the largest variance in the data.
* **t-Distributed Stochastic Neighbor Embedding (t-SNE)**: Focuses on preserving local structure (neighborhoods) by mapping high-dimensional data to a lower-dimensional space while maintaining relative distances between nearby points.

Global vs. Local Structure Preservation in Dimension Reduction

Principal component analysis (PCA) is a method of unsupervised learning techniques that functions as a method for dimension reduction in regression models.  

As explained in the textbook, the objective of PCA is *to find a low-dimensional representation of the observations that explain a good fraction of the variance* (ISLR).


University of Michigan - Ann Arbor

Unsupervised machine learning methods find structure in unlabeled data.
They are used when we DON'T have prior knowledge about the categories.

Unsupervised statistical learning

- Principal Component Analysis (PCA)
- Linear Discriminant Analysis (LDA)
- Generalized Discriminant Analysis (GDA)
- Singular Value Decomposition (SVD)

Methods of dimensionality reduction

Feature learning can be either supervised or unsupervised.

In supervised feature learning, features are learned using labeled input data. Examples include supervised neural networks, multilayer perceptron and (supervised) dictionary learning.

In unsupervised feature learning, features are learned with unlabeled input data. Examples include dictionary learning, independent component analysis, autoencoders, matrix factorization and various forms of clustering.


Methods of Feature Learning

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Which of the following are use-cases of supervised learning?

Principal Components Analysis (PCA)

Unlike supervised learning, unsupervised learning is used as part of exploratory data analysis (EDA) which aims to depict the main characteristics of a data set. However, it is difficult to use cross-validation in order to check the validity of our results when using unsupervised learning methods. This is because unlike supervised learning, which can check the accuracy of a predictive model by using it on data not used in fitting the model, unsupervised learning does not have any output values to match to, therefore making it difficult to assess the results since we don’t know the ‘true’ answer.

The Challenge of Unsupervised Learning

Types of unsupervised learning problems

Which ones are true about Supervised statistical learning?

Association rules allow you to establish associations amongst data objects inside large databases. This unsupervised technique is about discovering interesting relationships between variables in large databases. For example, people that buy a new home most likely to buy new furniture.

Association

T-SNE is an unsupervised machine learning method that is used to visualize the higher dimensional data in low dimensions. T-SNE is used for designing/implementation and can bring down any number of feature space into 2-D feature space.

T-Distributed Stochastic Neighbour Embedding (T-SNE)

Clustering tries to place unlabeled samples into groups based on similar characteristics. 
An example, these samples can be distinguished into 5 categories by some observed characteristic. Let's say we are clustering textbook samples by different school subjects (math, science, history, english, art). 
![The image from the cited reference](https://firebasestorage.googleapis.com/v0/b/onecademy-1.appspot.com/o/UploadedImages%2FMadeline-Rosenberg_Wed%2C%2019%20Feb%202020%2015%3A27%3A54%20GMT.png?alt=media&token=5b890a1d-803b-4ad5-a65f-1ec4043225c2) 

Clustering, an unsupervised statistical learning method

The primary advantages of unsupervised learning include:
- It identifies unknown patterns within data.
- It discovers features that are useful for categorization.
- It can operate in real-time, allowing input data to be analyzed dynamically.
- It utilizes unlabeled data, which is more abundant and easier to obtain than labeled data since it requires no manual intervention.

Advantages of Unsupervised Learning

This methodology is frequently used in recommendation problems. The examples  include deciding which stories to show on social media feed, which movies to suggest to user and many more.  In simple terms **collaborative filtering** finds what particular user liked and tries to find users with similar preferences and based on this information it would recommend to the current user the items that these other users liked. 

Collaborative Filtering

Independent component analysis (ICA) is a statistical and computational technique for revealing hidden factors that underlie sets of random variables, measurements, or signals. ICA defines a generative model for the observed multivariate data, which is typically given as a large database of samples.

Independent Component Analysis

Clustering: Clustering is the process of grouping the given data into different clusters or groups. 

Possible applications are :

1) e-commerce websites using clustering algorithms to implement a user-specific recommendation system.
2) Video recommendation by detecting groups of similar subscribers, 

Commonly used algorithms are:

1)K-Means
2)Expectation Maximization
3)Hierarchical Cluster Analysis (HCA)

(https://www.pythonistaplanet.com/applications-of-unsupervised-learning/)

Real-World Applications Of Unsupervised Learning

A type of linear factor model that uses the slowness principle and applies a cost to time. For features that may not change rapidly over time, e.g. whether or not an animal is in frame of a video, slow feature analysis can smooth the detection of those features. A cost may be added of the following form:
$$\lambda \sum_{t} L(f(x^{(t+1)}), f(x^{(t)}))  $$
It has not been used frequently or in any state-of-the-art applications, but may hold promise for more interesting system to come.  

Slow Feature Analysis

Linear Factor Models Lecture - Berkeley

Slow feature analysis is motivated by a general principle called the slowness principle. The idea is that the important characteristics of scenes change very slowly compared to the individual measurements that make up a description of a scene.  The slowness principal may be introduced by adding a term to the cost function measuring the difference between feature extractors according to time. 
The SFA algorithm consists of defining $f(x; \theta)$ to be a linear transformation and solving the optimization problem:
$min_{\theta}E_t (f(x^{t+1})_i-f(x^{t})_i)^2$
subject to the constraints:
$E_t (f(x^{t})_i)=0$
and
$E_t [f(x^{t})^2_i]=1$

Slow feature analysis

Linear discriminant analysis (LDA) is another means of classification similar to logistic regression, but better suited to handle non-binary classifications (i.e. when there are >=3 possible output classes).  

LDA assumes Gaussian distributions for observations in each class, with means being class specific and a covariance common to all classes. Then with the help of Bayes' theorem, the probability for one observation being in each class can be estimated. The class with the highest probability would be the classification result.

In this assumption, the classification is equivalent to maximize the *discriminant function* 
$$
\delta_{k}(x)=x^{T} \boldsymbol{\Sigma}^{-1} \mu_{k}-\frac{1}{2} \mu_{k}^{T} \boldsymbol{\Sigma}^{-1} \mu_{k}+\log \pi_{k}
$$
where ${\mu}_{k}$ is the mean for observations in the $k$th class, ${\pi}_{k}$ is the prior probability that an observation belongs to the $k$th class, and $\Sigma$ is the common covariance matrix. The *discriminant function* is linear to $x$, which is why the method is called *linear discriminant analysis*.


Linear Discriminant Analysis (LDA)

Similar to Linear discriminant analysis (LDA), a Generalized Discriminant Analysis (GDA) method project the data in the high-dimensional space to a space of fewer dimensions. 
LDA method does not work for nonlinear problems. By mapping the input space into a high dimensional feature space with linear properties, a Generalized Discriminant Analysis (GDA) can deal with nonlinear discriminant analysis.
The mapping process of GDA is similar to the mapping process in Support Vector Machine (SVM).


Generalized discriminant analysis (GDA)

Singular value decomposition (SVD) is an unsupervised dimensionality reduction technique used for feature extraction.  The singular value decomposition of an m x n matrix M is a factorization of the form U$$\sum$$M*.  U is an m x m unitary matrix. $$\sum$$ is a m x n diagonal matrix. V is a v x v unitary matrix.
Image link: [www.wikiwand.com/en/Singular_value_decomposition](https://www.wikiwand.com/en/Singular_value_decomposition)
 

Singular value decomposition (SVD)

Deep learning is a particular kind of machine learning that achieves great power and flexibility by representing the world as a nested hierarchy of concepts, with each concept defined in relation to simpler concepts, and more abstract representationss computed in terms of less abstract ones.


Learn Before

Related

Learn After