Different methods are required for fitting a model for classification. Two reasons:
1. If there are >2 classifications, simply assigning dummy numerical variables and making a prediction based on the nearest classifier would imply an ordering to the classifications; that is to say that classification 4 and 3 are more similar than 4 and 1. 
2. Even using a binary classification, 1 or 0, an OLS model can produce an output outside of the range [0, 1], which is difficult to interpret and again implies an ordering to the classifications

OLS fitting cannot be used for classification

LDA is more stable than logistic regression in instances where the classes are well separated or when "n is small and the distribution of the predictors X is approximately normal in each of the classes" (ISLR). 

Using LDA vs Logistic Regression

This video series breaks down logistic regression and its applications.

[Logistic Regression]. (2020, February 21). Logistic Regression [Video File]. Retrieved from [www.youtube.com/playlist?list=PLblh5JKOoLUKxzEP5HA2d-Li7IJkHfXSe](https://www.youtube.com/playlist?list=PLblh5JKOoLUKxzEP5HA2d-Li7IJkHfXSe) 

Logistic Regression Videos

There are three out of four binary classification metrics mentioned in the Book of Why; true positive, false positive, and false negative.  The fourth one that is not mentioned in the book is true negative.  The following are the two primary building blocks of such four metrics: 

1. True vs. False: These are about prediction's correctness whether a prediction outcome was made accurately/ correctly.  The correct prediction is True, and the incorrect prediction is False.
2. Positive vs. Negative: These are about a binary classification -(e.g.) In the mammogram example in the book, "positive" is positive test results (breast cancer), whereas "negative" is negative test results (not breast cancer).  

The combinations of the above concepts result in True positive, True negative, false positive, and false negative, each of which is explained in the subsequent nodes to follow.  


Binary Classification Metrics

The hypothesis we are using for the logistic regression is the sigmoid function:



Hypothesis

The hypothesis function for the logistic regression is the sigmoid function:


Hypothesis function

Given X, we want $\hat{y}=P(y = 1|X)$   $0 \leq \hat{y} \leq 1$

Parameters: $w \in R^{n_{x}}, b \in R$

Output (Activation): $\hat{y}^{(i)} = a^{(i)} = \sigma (w^{T}x^{(i)} + b)$

Given $[(x^{(1)}, y^{(1)}), (x^{(2)}, y^{(2)}), ..., (x^{(m)}, y^{(m)})]$, we want $\hat{y}^{i} \approx y^{i}$

$\sigma(z^{(i)}) = \frac{1}{1 + e^{-z^{(i)}}}$

If z is large positive, $\sigma(z^{(i)}) \approx \frac{1}{1 + 0} = 1$

If z is large negative, $\sigma(z^{(i)}) \approx \frac{1}{1 + Big Num} \approx 0$

Logistic Regression Formulation

 $$P(\hat{Y})=e^{(\beta_{0}+\beta_{1}X)}/[1+e^{(\beta_{0}+\beta_{1}X)}]$$
$$\Rightarrow P(\hat{Y})+P(\hat{Y})\times[e^{(\beta_{0}+\beta_{1}X)}]=e^{(\beta_{0}+\beta_{1}X)}$$
$$\Rightarrow P(\hat{Y})=[1-P(\hat{Y})]\times [e^{(\beta_{0}+\beta_{1}X)}]$$
$$\Rightarrow P(\hat{Y})/(1-P(\hat{Y}))=e^{(\beta_{0}+\beta_{1}X)} (1.1)$$
The left part is called *odds ratio*.
$$\Rightarrow\log{(P(X)/(1-P(X)))}=\beta_{0}+\beta_{1}X$$
The left hand side is called *log odds ratio*.
This is classified under linear regressions because the right hand side is the basic formula for Linear Regression.
Coefficient $\beta_{1}$ means: one-unit increase in x is associated with an increase in the log odds of 0/1 by $\beta_{1}$ units.

Logistic Regression Mathematical Equation

Like ridge and lasso regression, a regularization penalty on the model coefficients can also be applied with logistic regression, and is controlled with the parameter C. 

In fact, the same L2 regularization penalty used for ridge regression is turned on by default for logistic regression with a default value C = 1. With large values of C, logistic regression tries to fit the training data as well as possible. While with small values of C, the model tries harder to find model coefficients that are closer to 0, even if that model fits the training data a little bit worse. 

If the L1 and L2 norms are added to the logistic regression, the effect is: feature selection can be done, and overfitting can also be prevented to a certain extent. Reason: L1–LASSO can generate sparse solutions for feature selection; L2–Ridge constrains model parameters to prevent overfitting. In addition, L2 can get smooth weights.

Logistic Regression - Regularization

Linear regression is used for prediction and logical regression is used for classification. Linear regression is to fit the function, and logical regression is to predict the function. Linear regression uses least square method to calculate parameters, and logical regression uses maximum likelihood estimation to calculate parameters. Linear regression is more susceptible to outliers, while logical regression has better stability to outliers.


Linear Regression vs Logistic Regression

Softmax regression is the generalization of logistic regression to multiple classes. In other words, each data point belongs to one of multiple classes (rather than just two options, as is the case for logistic regression). Hence softmax regression is also called multi-class logistic regression or multinomial logistic regression.

Softmax Regression (Activation)

Logistic regression is used to model discontinuous output choices - for example, when a variable is 'yes' or 'no' rather than a continuous value. In this case, logistic regression calculates the probability of a given categorical output, and decides on a threshold at which the model decides between categories to select for output. 

University of Michigan - Ann Arbor

- Logistic Regression
- Linear Discriminat Analysis
- K-nearest Neighbors

Most Common Classifiers

Linear regression is a linear approach to model a target prediction value based on independent variables.

Linear Regression

Binary classification is the simplest form of classification problem where there are exactly two possible classes or categories to predict. A common example is designing a classifier that takes an image as input and outputs the predicted class assignment from a set of two options, such as $$\textrm{\{cat, dog\}}$$.

Binary Classification

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Logistic Regression

Linear discriminant analysis (LDA) is another means of classification similar to logistic regression, but better suited to handle non-binary classifications (i.e. when there are >=3 possible output classes).  

LDA assumes Gaussian distributions for observations in each class, with means being class specific and a covariance common to all classes. Then with the help of Bayes' theorem, the probability for one observation being in each class can be estimated. The class with the highest probability would be the classification result.

In this assumption, the classification is equivalent to maximize the *discriminant function* 
$$
\delta_{k}(x)=x^{T} \boldsymbol{\Sigma}^{-1} \mu_{k}-\frac{1}{2} \mu_{k}^{T} \boldsymbol{\Sigma}^{-1} \mu_{k}+\log \pi_{k}
$$
where ${\mu}_{k}$ is the mean for observations in the $k$th class, ${\pi}_{k}$ is the prior probability that an observation belongs to the $k$th class, and $\Sigma$ is the common covariance matrix. The *discriminant function* is linear to $x$, which is why the method is called *linear discriminant analysis*.


Linear Discriminant Analysis (LDA)

K-nearest neighbors is a non-parametric statistical method, which predicts based on the *k* nearest neighbors. k-NN classifiers are an example of what's called instance based or memory based supervised learning.

It can be applied to both classification and regression problems.
- When applied to classification problems, the prediction result would be the most common one among the *k* nearest training observations.
- When applied to regression problems, the prediction result would be the average of the *k* nearest training observations.

Some key notes about this method:
- It makes few assumptions about structure of data
- It gives potentially accurate but sometimes unstable predictions 
- It is sensitive to small changes in training data



K-nearest Neighbors (k-NN)

Based on the number of independent variables used, there are two types of linear regression:
- Simple Linear Regression
- Multiple Linear Regression

Types of linear regression

The terms of a linear model can be one of the following:
	
	- The Constant
	- A parameter multiplied by an independent variable (I.V)


Then, you build the equation by only adding the terms together. These rules limit the form to just one type:
Dependent variable = constant + parameter * IV + … + parameter * IV

Nonlinear regression models don’t fit this type.

Examples of nonlinear models include: 
	

When is a model considered linear?

This is a series of videos that introduces linear regression models and breaks down the process of linear regression.

[StatQuest with Josh Starmer]. (2020, February 21). Linear Regression and Linear Models [Video File]. Retrieved from [www.youtube.com/playlist?list=PLblh5JKOoLUIzaEkCLIUxQFjPIlapw8nU](https://www.youtube.com/playlist?list=PLblh5JKOoLUIzaEkCLIUxQFjPIlapw8nU) 

Linear Regression and Linear Models Videos

Here is an example of using R to do a simple linear regression. 
```R
#MASS package is a very large collection of data sets and #functions
library(MASS) 

#ISLR package includes the data sets associated with this #book
library(ISLR) 

lm.fit=lm(medv∼lstat,data=Boston) 
# medv as a response, lstat as a predictor

summary(lm.fit) # gives us pvalues and standard errors for the coeﬃcients, as well as the R2 statistic and F-statistic for the model

confint(lm.fit) # obtain a confidence interval

predict (lm.fit ,data.frame(lstat=c(5,10,15)), interval="confidence") #prediction in confidence intervals

predict (lm.fit ,data.frame(lstat=c(5,10,15)), interval="prediction") #prediction in prediction intervals

attach(Boston) # attach data to Boston

plot(lstat ,medv) 
abline(lm.fit ,lwd=3,col="red") 
# draw out the regression line
```
Expected output (plot and abline only):
![The expected output from the plot and abline commands](https://firebasestorage.googleapis.com/v0/b/onecademy-1.appspot.com/o/UploadedImages%2FWenfei-Tang_Wed%2C%2026%20Feb%202020%2020%3A51%3A58%20GMT.png?alt=media&token=5cf41866-c6fe-4732-88c5-2233fe889aa3) 

Example of Linear Regression Using R

We estimate the parameters $\beta_0, \beta_1, \beta_2, \dots, \beta_p$ such that we minimize the error in the following approximation:
$$Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots + \beta_p X_p + \epsilon$$

Training or Fitting a linear regression model

The extent to which the linear regression model fits the data, the quality of linear regression fit, can be achieved by:
1. $ R^2 $
2. Residual Standard Error
3. Goodnesss of Fit
4. F Statistics
5. Mean Square Error (MSE)

Assessing the accuracy of linear regression

Linear regression is an example of a parametric method because it assumes a linear functional form for f(X). But if the specified functional form is far from the truth, and prediction accuracy is our goal, then the parametric method will perform poorly. This leads to higher bias. There is also a well known statistical theory behind linear regression.

KNN is an example of non parametric method that uses feature similarity. It is pretty useful, because in the real world, most of the practical data does not obey the typical theoretical assumptions made (eg gaussian mixtures, linearly separable etc). Non-parametric methods do not explicitly assume a parametric form for f(X), and thereby provide an alternative and more flexible approach for performing regression. This means it will have less bias and is more robust with highly non-linear settings. However, KNN regression is not as widely studied as linear regression.

Comparison of Linear Regression with K-Nearest Neighbors

- Linearity of the response-predictor relations
- Normal distribution of residuals
-  Error terms, e1,e2,...e3, are uncorrelated
-  Error terms have a constant variance
-  Little or no Multicollinearity between the features
-  Homoscedasticity Assumption
-  Large sample size and few or no outliers

Assumptions of Linear Regression

The regression coefficient is $$\beta_1$$, in the regression equation $$y \approx \beta_0 + \beta_1 X$$, is an esitmate of the rate of change (slope) between X and Y.

Regression Coefficient

It is easier to implement, interpret, and infer from the estimation of linear regression. However, one can rarely find a perfect linear relationship in the real world. Estimation using nonlinear regression is more difficult to implement, interpret, and infer from, but it gives us more predictive power by relaxing the assumption of linearity of the model.

Linear regression vs. nonlinear regression

- L1 Regularization
- L2 Regularization
- Dropout Regularization
- Data Augmentation
- Early Stopping
- Tangent Distance
- Tangent Prop
- Manifold Tangent Classifier

Popular Regularization Techniques in Deep Learning

We can use  polynomial future transformation to transform a problem into a higher dimensional regression space. Adding these extra polynomial features allows us a much richer set of complex functions that we can use to fit to the data. This is like allowing polynomials to be fit to the training data instead of simply a straight line, but using the same least-squares criterion that minimizes mean squared error. This approach of adding new features like polynomial features is also very effective with classification.

Polynomial Future Transformation

To make up for the under-fitting or over-fitting problems of ordinary linear regression models, locally weighted linear regression introduce weights to loss function. 
Its loss is transmitted to:
$J(\theta)=\sum_{i=1}^m w^i (y^{(i)} - \theta^T x^{(i)})^2$
w is used to represent the weights. Its value is calculated using Gaussian kernel function:
$w^i = exp(\frac{(x-x^i)^2}{-2k^2})$

Locally Weighted Linear Regression

In the context of linear regression, specific mathematical notation is used to systematically describe the dataset. Typically, $$n$$ denotes the total number of examples. Superscripts are utilized to enumerate individual samples and their corresponding targets, while subscripts are used to index their specific coordinates or features. For instance, $$\mathbf{x}^{(i)}$$ represents the complete $$i^{\textrm{th}}$$ sample, and $$x_j^{(i)}$$ designates the $$j^{\textrm{th}}$$ coordinate of that $$i^{\textrm{th}}$$ sample.

Linear Regression Dataset Notation

The foundational concepts of linear regression date back to the early 19th century, with significant contributions from mathematicians Carl Friedrich Gauss in 1809 and Adrien-Marie Legendre in 1805. Despite its historical age, linear regression remains one of the simplest and most popular standard tools for tackling modern regression problems.

History of Linear Regression

In the simplest case of one-dimensional inputs, a linear regression model attempts to fit a straight line through the observed data points. The fitness of this line depends purely on the choice of model parameters (the weight and the bias), which are adjusted to minimize the distance between the line's predictions and the true target values.

Linear Regression One-Dimensional Fit

The squared error is the standard loss function for regression problems, used to quantify the discrepancy between a single predicted value $$\hat{y}^{(i)}$$ and its true corresponding label $$y^{(i)}$$. The loss for a single example is defined mathematically as $$l^{(i)}(\mathbf{w}, b) = \frac{1}{2} \left(\hat{y}^{(i)} - y^{(i)}ight)^2$$. The quadratic form heavily penalizes large differences, and the constant fraction $$\frac{1}{2}$$ is conventionally added because it cleanly cancels out when taking the derivative during optimization.

Squared Error Loss

A fundamental assumption of linear regression is that the relationship between the features $$\mathbf{x}$$ and the target $$y$$ is approximately linear. This means the expected value of the target, defined as the conditional mean $$E[Y \mid X=\mathbf{x}]$$, can be mathematically expressed as a weighted sum of the input features $$\mathbf{x}$$. This framework allows the actual target values to deviate from their expected conditional mean due to observation noise.

Linear Regression Conditional Mean Assumption

In a multiple-feature linear regression model, weights are parameters that determine the influence of each individual feature on the target prediction. When the model is expressed as a weighted sum, such as $$\hat{y} = w_1 x_1 + \dots + w_d x_d + b$$, each weight quantifies the specific contribution of its corresponding feature. In compact linear algebra notation, these parameters are collected into a weight vector $$\mathbf{w} \in \mathbb{R}^d$$.

Linear Regression Weight Parameters

In a linear regression model, the bias (also referred to as an offset or intercept) is a parameter that determines the baseline value of the model's estimate when all input features are exactly zero. The addition of the bias term $$b$$ is mathematically necessary because it permits the model to express general linear functions as an affine transformation, rather than restricting predictions to lines that must pass directly through the origin.

Linear Regression Bias Parameter

Linear regression can be conceptualized as a single-layer, fully connected neural network. In this representation, each given input feature $$x_1, \ldots, x_d$$ corresponds to an input neuron. Since the goal is to predict a single numerical value, all input neurons are directly connected to a single computed output neuron $$o_1$$. The total number of inputs $$d$$ is referred to as the feature dimensionality in the input layer.

Linear Regression as a Neural Network

To evaluate machine learning models, we often generate synthetic datasets where the underlying ground truth relationship is known. For a linear regression task, we can draw a design matrix $$\mathbf{X}$$ of features from a standard normal distribution. The corresponding labels $$\mathbf{y}$$ are computed by applying a ground truth linear function defined by true weights $$\mathbf{w}$$ and bias $$b$$, and then corrupting the output with additive noise $$\boldsymbol{\epsilon}$$ drawn from a normal distribution with mean $$\mu=0$$ and standard deviation $$\sigma = 0.01$$: $$\mathbf{y}= \mathbf{X} \mathbf{w} + b + \boldsymbol{\epsilon}$$. This procedure ensures that the generated labels simulate realistically observed data containing inherent random variation.

Synthetic Data Generation for Linear Regression

- Email spam detection (spam or not).
- Churn prediction (churn or not).
- Conversion prediction (buy or not).

Examples of Binary Classification

Logistic regression is a statistical model that forces the output of an otherwise unbounded function to the interval (0,1).

$$p(y=1 | x;\theta) = \sigma(\theta^\top x) $$
$$\sigma(x) = \frac{e^x}{e^x + 1}$$
$$p(y=0 | x;\theta) = 1 - p(y=1 | x;\theta)$$

Where $\theta$ is the parameter vector.

For binary classification problems, logistic regression can be used instead of linear regression to handle the discontinuous nature of the target output.

However, the sigmoid function $\sigma$ has no local minimum or maximum, so a technique like gradient decent must be used instead.

Learn Before

Related

Learn After