In QDA, we need to estimate Σ𝑘 for each class 𝑘∈{1,…,𝐾} rather than assuming Σ𝑘=Σ as in LDA. The discriminant function of LDA is quadratic in 𝑥 in the image below.

Since QDA estimates a covariance matrix for each class, it has a greater number of effective parameters than LDA. We can derive the number of parameters in the following way.
- We need 𝐾 class priors 𝜋𝑘. Since ∑𝐾𝑖=1𝜋𝑘=1, we do not need a parameter for one of the priors. Thus, there are 𝐾−1 free parameters for the priors.
- Since there are 𝐾 centroids, 𝜇𝑘, with 𝑝 entries each, there are 𝐾𝑝 parameters relating to the means.
- From the covariance matrix, Σ𝑘, we only need to consider the diagonal and the upper right triangle. This region of the covariance matrix has 𝑝(𝑝+1)2 elements. Since 𝐾 such matrices need to be estimated, there are 𝐾𝑝(𝑝+1)2 parameters relating to the covariance matrices.

Thus, the effective number of QDA parameters is 𝐾−1+𝐾𝑝+𝐾𝑝(𝑝+1)2.

QDA Formula

QDA is a variant of LDA in which an individual covariance matrix is estimated for every class of observations. QDA is particularly useful if there is prior knowledge that individual classes exhibit distinct covariances. A disadvantage of QDA is that it cannot be used as a dimensionality reduction technique.

University of Michigan - Ann Arbor

Linear discriminant analysis (LDA) is another means of classification similar to logistic regression, but better suited to handle non-binary classifications (i.e. when there are >=3 possible output classes).  

LDA assumes Gaussian distributions for observations in each class, with means being class specific and a covariance common to all classes. Then with the help of Bayes' theorem, the probability for one observation being in each class can be estimated. The class with the highest probability would be the classification result.

In this assumption, the classification is equivalent to maximize the *discriminant function* 
$$
\delta_{k}(x)=x^{T} \boldsymbol{\Sigma}^{-1} \mu_{k}-\frac{1}{2} \mu_{k}^{T} \boldsymbol{\Sigma}^{-1} \mu_{k}+\log \pi_{k}
$$
where ${\mu}_{k}$ is the mean for observations in the $k$th class, ${\pi}_{k}$ is the prior probability that an observation belongs to the $k$th class, and $\Sigma$ is the common covariance matrix. The *discriminant function* is linear to $x$, which is why the method is called *linear discriminant analysis*.


Linear Discriminant Analysis (LDA)

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

LDA is more stable than logistic regression in instances where the classes are well separated or when "n is small and the distribution of the predictors X is approximately normal in each of the classes" (ISLR). 

Using LDA vs Logistic Regression

According to Bayes' theorem, we have
$$
P\left(Y=k | X=x\right)=\frac{P\left(Y=k\right) P\left(X=x | Y=k\right)}{\sum_{l=1}^{K} P\left(Y=l\right) P\left(X=x | Y=l\right)}\\
=\frac{\pi_{k} f_{k}(x)}{\sum_{l=1}^{K} \pi_{l} f_{l}(x)}.
$$
In LDA with the assumption of Gaussian distribution in every class, we have the Gaussian density 
$$
f_k(x)=\frac{1}{(2 \pi_k)^{p / 2}|\boldsymbol{\Sigma}|^{1 / 2}} \exp \left(-\frac{1}{2}(x-\mu_k)^{T} \boldsymbol{\Sigma}^{-1}(x-\mu_k)\right)
$$
for a $p$-dimensional random vector $X$ with the distribution $N(\mu_k, \mathbf{\Sigma})$. All we need to do is to plug $f_k(x)$ into Bayes' theorem above and find out the $k$ which maximize $P\left(Y=k | X=x\right)$. LDA is a Bayes classifier with the assumption of Gaussian distribution in every class.

Using Bayes' theorem in LDA

First, the means and covariance of the assumed gaussian distributions in each class need to be estimated. When $p=1$, which means only one predictor, that is to estimate the class specific mean $\mu_k$ and the common variance $\sigma^2$. They can be estimated as follows
$$\hat{\mu}_{k}=\frac{1}{n_{k}} \sum_{i: y_{i}=k} x_{i}$$
$$\hat{\sigma}^{2}=\frac{1}{n-K} \sum_{k=1}^{K} \sum_{i: y_{i}=k}\left(x_{i}-\hat{\mu}_{k}\right)^{2}$$
where $n$ is the total number of training observations, and $n_k$ is the number
of training observations in the kth class. When $p>1$, which means multiple predictors, the estimation would be similar but much more complicated with $p(p+1)/2$ parameters in the covariance matrix.

As for the prior probabilty $\pi_k$, it can be estimated based on the proportion of class $k$ observations in the training set, which would be 
$$
\hat{\pi}_{k}=n_{k} / n.
$$


Estimate parameters in LDA

LDA assumes that the covariance matrix is common to all classes, while QDA assumes class-specific covariance matrices.  Notice that for a $p$-dimensional observation $X$, one covariance needs $p(p+1)/2$ parameters, and then QDA will need $Kp(p+1)/2$ parameters for the covariance matrices. While this makes QDA much more flexible, the much more parameters also bring high variance. QDA is preferred when the training set is large, and so the variance is not a major concern.

Using LDA vs QDA

Quadratic Discriminant Analysis (QDA)

Both rank the new axes in the order of importance. PC1 (the first new axis that PCA creates) accounts for the most variation in data, PC2 (the second new axes) does the second-best job and so on… LD1 (the first new axis that LDA creates) accounts for the most variation in data, LD2(the second new axes) does the second-best job and so on…
Both the algorithms tell us which attribute or feature is contributing more in creating the new axes.
LDA is like PCA — both try to reduce the dimensions. PCA looks for attributes with the most variance. LDA tries to maximize the separation of known categories.

Learn Before

Related

Learn After