LDA is more stable than logistic regression in instances where the classes are well separated or when "n is small and the distribution of the predictors X is approximately normal in each of the classes" (ISLR). 

University of Michigan - Ann Arbor

Logistic regression is used to model discontinuous output choices - for example, when a variable is 'yes' or 'no' rather than a continuous value. In this case, logistic regression calculates the probability of a given categorical output, and decides on a threshold at which the model decides between categories to select for output. 

Logistic Regression

Linear discriminant analysis (LDA) is another means of classification similar to logistic regression, but better suited to handle non-binary classifications (i.e. when there are >=3 possible output classes).  

LDA assumes Gaussian distributions for observations in each class, with means being class specific and a covariance common to all classes. Then with the help of Bayes' theorem, the probability for one observation being in each class can be estimated. The class with the highest probability would be the classification result.

In this assumption, the classification is equivalent to maximize the *discriminant function* 
$$
\delta_{k}(x)=x^{T} \boldsymbol{\Sigma}^{-1} \mu_{k}-\frac{1}{2} \mu_{k}^{T} \boldsymbol{\Sigma}^{-1} \mu_{k}+\log \pi_{k}
$$
where ${\mu}_{k}$ is the mean for observations in the $k$th class, ${\pi}_{k}$ is the prior probability that an observation belongs to the $k$th class, and $\Sigma$ is the common covariance matrix. The *discriminant function* is linear to $x$, which is why the method is called *linear discriminant analysis*.


Linear Discriminant Analysis (LDA)

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Different methods are required for fitting a model for classification. Two reasons:
1. If there are >2 classifications, simply assigning dummy numerical variables and making a prediction based on the nearest classifier would imply an ordering to the classifications; that is to say that classification 4 and 3 are more similar than 4 and 1. 
2. Even using a binary classification, 1 or 0, an OLS model can produce an output outside of the range [0, 1], which is difficult to interpret and again implies an ordering to the classifications

OLS fitting cannot be used for classification

Using LDA vs Logistic Regression

This video series breaks down logistic regression and its applications.

[Logistic Regression]. (2020, February 21). Logistic Regression [Video File]. Retrieved from [www.youtube.com/playlist?list=PLblh5JKOoLUKxzEP5HA2d-Li7IJkHfXSe](https://www.youtube.com/playlist?list=PLblh5JKOoLUKxzEP5HA2d-Li7IJkHfXSe) 

Logistic Regression Videos

There are three out of four binary classification metrics mentioned in the Book of Why; true positive, false positive, and false negative.  The fourth one that is not mentioned in the book is true negative.  The following are the two primary building blocks of such four metrics: 

1. True vs. False: These are about prediction's correctness whether a prediction outcome was made accurately/ correctly.  The correct prediction is True, and the incorrect prediction is False.
2. Positive vs. Negative: These are about a binary classification -(e.g.) In the mammogram example in the book, "positive" is positive test results (breast cancer), whereas "negative" is negative test results (not breast cancer).  

The combinations of the above concepts result in True positive, True negative, false positive, and false negative, each of which is explained in the subsequent nodes to follow.  


Binary Classification Metrics

The hypothesis we are using for the logistic regression is the sigmoid function:



Hypothesis

The hypothesis function for the logistic regression is the sigmoid function:


Hypothesis function

Given X, we want $\hat{y}=P(y = 1|X)$   $0 \leq \hat{y} \leq 1$

Parameters: $w \in R^{n_{x}}, b \in R$

Output (Activation): $\hat{y}^{(i)} = a^{(i)} = \sigma (w^{T}x^{(i)} + b)$

Given $[(x^{(1)}, y^{(1)}), (x^{(2)}, y^{(2)}), ..., (x^{(m)}, y^{(m)})]$, we want $\hat{y}^{i} \approx y^{i}$

$\sigma(z^{(i)}) = \frac{1}{1 + e^{-z^{(i)}}}$

If z is large positive, $\sigma(z^{(i)}) \approx \frac{1}{1 + 0} = 1$

If z is large negative, $\sigma(z^{(i)}) \approx \frac{1}{1 + Big Num} \approx 0$

Logistic Regression Formulation

 $$P(\hat{Y})=e^{(\beta_{0}+\beta_{1}X)}/[1+e^{(\beta_{0}+\beta_{1}X)}]$$
$$\Rightarrow P(\hat{Y})+P(\hat{Y})\times[e^{(\beta_{0}+\beta_{1}X)}]=e^{(\beta_{0}+\beta_{1}X)}$$
$$\Rightarrow P(\hat{Y})=[1-P(\hat{Y})]\times [e^{(\beta_{0}+\beta_{1}X)}]$$
$$\Rightarrow P(\hat{Y})/(1-P(\hat{Y}))=e^{(\beta_{0}+\beta_{1}X)} (1.1)$$
The left part is called *odds ratio*.
$$\Rightarrow\log{(P(X)/(1-P(X)))}=\beta_{0}+\beta_{1}X$$
The left hand side is called *log odds ratio*.
This is classified under linear regressions because the right hand side is the basic formula for Linear Regression.
Coefficient $\beta_{1}$ means: one-unit increase in x is associated with an increase in the log odds of 0/1 by $\beta_{1}$ units.

Logistic Regression Mathematical Equation

Like ridge and lasso regression, a regularization penalty on the model coefficients can also be applied with logistic regression, and is controlled with the parameter C. 

In fact, the same L2 regularization penalty used for ridge regression is turned on by default for logistic regression with a default value C = 1. With large values of C, logistic regression tries to fit the training data as well as possible. While with small values of C, the model tries harder to find model coefficients that are closer to 0, even if that model fits the training data a little bit worse. 

If the L1 and L2 norms are added to the logistic regression, the effect is: feature selection can be done, and overfitting can also be prevented to a certain extent. Reason: L1–LASSO can generate sparse solutions for feature selection; L2–Ridge constrains model parameters to prevent overfitting. In addition, L2 can get smooth weights.

Logistic Regression - Regularization

Linear regression is used for prediction and logical regression is used for classification. Linear regression is to fit the function, and logical regression is to predict the function. Linear regression uses least square method to calculate parameters, and logical regression uses maximum likelihood estimation to calculate parameters. Linear regression is more susceptible to outliers, while logical regression has better stability to outliers.


Linear Regression vs Logistic Regression

Softmax regression is the generalization of logistic regression to multiple classes. In other words, each data point belongs to one of multiple classes (rather than just two options, as is the case for logistic regression). Hence softmax regression is also called multi-class logistic regression or multinomial logistic regression.

Softmax Regression (Activation)

According to Bayes' theorem, we have
$$
P\left(Y=k | X=x\right)=\frac{P\left(Y=k\right) P\left(X=x | Y=k\right)}{\sum_{l=1}^{K} P\left(Y=l\right) P\left(X=x | Y=l\right)}\\
=\frac{\pi_{k} f_{k}(x)}{\sum_{l=1}^{K} \pi_{l} f_{l}(x)}.
$$
In LDA with the assumption of Gaussian distribution in every class, we have the Gaussian density 
$$
f_k(x)=\frac{1}{(2 \pi_k)^{p / 2}|\boldsymbol{\Sigma}|^{1 / 2}} \exp \left(-\frac{1}{2}(x-\mu_k)^{T} \boldsymbol{\Sigma}^{-1}(x-\mu_k)\right)
$$
for a $p$-dimensional random vector $X$ with the distribution $N(\mu_k, \mathbf{\Sigma})$. All we need to do is to plug $f_k(x)$ into Bayes' theorem above and find out the $k$ which maximize $P\left(Y=k | X=x\right)$. LDA is a Bayes classifier with the assumption of Gaussian distribution in every class.

Using Bayes' theorem in LDA

First, the means and covariance of the assumed gaussian distributions in each class need to be estimated. When $p=1$, which means only one predictor, that is to estimate the class specific mean $\mu_k$ and the common variance $\sigma^2$. They can be estimated as follows
$$\hat{\mu}_{k}=\frac{1}{n_{k}} \sum_{i: y_{i}=k} x_{i}$$
$$\hat{\sigma}^{2}=\frac{1}{n-K} \sum_{k=1}^{K} \sum_{i: y_{i}=k}\left(x_{i}-\hat{\mu}_{k}\right)^{2}$$
where $n$ is the total number of training observations, and $n_k$ is the number
of training observations in the kth class. When $p>1$, which means multiple predictors, the estimation would be similar but much more complicated with $p(p+1)/2$ parameters in the covariance matrix.

As for the prior probabilty $\pi_k$, it can be estimated based on the proportion of class $k$ observations in the training set, which would be 
$$
\hat{\pi}_{k}=n_{k} / n.
$$


Estimate parameters in LDA

LDA assumes that the covariance matrix is common to all classes, while QDA assumes class-specific covariance matrices.  Notice that for a $p$-dimensional observation $X$, one covariance needs $p(p+1)/2$ parameters, and then QDA will need $Kp(p+1)/2$ parameters for the covariance matrices. While this makes QDA much more flexible, the much more parameters also bring high variance. QDA is preferred when the training set is large, and so the variance is not a major concern.

Using LDA vs QDA

QDA is a variant of LDA in which an individual covariance matrix is estimated for every class of observations. QDA is particularly useful if there is prior knowledge that individual classes exhibit distinct covariances. A disadvantage of QDA is that it cannot be used as a dimensionality reduction technique.

Quadratic Discriminant Analysis (QDA)

Both rank the new axes in the order of importance. PC1 (the first new axis that PCA creates) accounts for the most variation in data, PC2 (the second new axes) does the second-best job and so on… LD1 (the first new axis that LDA creates) accounts for the most variation in data, LD2(the second new axes) does the second-best job and so on…
Both the algorithms tell us which attribute or feature is contributing more in creating the new axes.
LDA is like PCA — both try to reduce the dimensions. PCA looks for attributes with the most variance. LDA tries to maximize the separation of known categories.

Learn Before

Related