Logistic regression is used to model discontinuous output choices - for example, when a variable is 'yes' or 'no' rather than a continuous value. In this case, logistic regression calculates the probability of a given categorical output, and decides on a threshold at which the model decides between categories to select for output. 

Logistic Regression

Linear discriminant analysis (LDA) is another means of classification similar to logistic regression, but better suited to handle non-binary classifications (i.e. when there are >=3 possible output classes).  

LDA assumes Gaussian distributions for observations in each class, with means being class specific and a covariance common to all classes. Then with the help of Bayes' theorem, the probability for one observation being in each class can be estimated. The class with the highest probability would be the classification result.

In this assumption, the classification is equivalent to maximize the *discriminant function* 
$$
\delta_{k}(x)=x^{T} \boldsymbol{\Sigma}^{-1} \mu_{k}-\frac{1}{2} \mu_{k}^{T} \boldsymbol{\Sigma}^{-1} \mu_{k}+\log \pi_{k}
$$
where ${\mu}_{k}$ is the mean for observations in the $k$th class, ${\pi}_{k}$ is the prior probability that an observation belongs to the $k$th class, and $\Sigma$ is the common covariance matrix. The *discriminant function* is linear to $x$, which is why the method is called *linear discriminant analysis*.


Linear Discriminant Analysis (LDA)

K-nearest neighbors is a non-parametric statistical method, which predicts based on the *k* nearest neighbors. k-NN classifiers are an example of what's called instance based or memory based supervised learning.

It can be applied to both classification and regression problems.
- When applied to classification problems, the prediction result would be the most common one among the *k* nearest training observations.
- When applied to regression problems, the prediction result would be the average of the *k* nearest training observations.

Some key notes about this method:
- It makes few assumptions about structure of data
- It gives potentially accurate but sometimes unstable predictions 
- It is sensitive to small changes in training data



K-nearest Neighbors (k-NN)

- Logistic Regression
- Linear Discriminat Analysis
- K-nearest Neighbors

University of Michigan - Ann Arbor

Classifiers are techniques used to predict qualitative variables.
- Methods that are not very computer intensive (widely-used)
- Methods that are much more computer intensive
- Decision Tree that can be used on regression and classification problems

Methods of Classification

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Most Common Classifiers

- Generalized Additive Models
- Trees
- Random Forests
- Boosting
- Support Vector Machines
- Naive Bayes Classifier

Learn Before

Related

Learn After