Residual Standard Error is an estimate of σ standard deviation. It assesses the quality of linear regression. It is the average deviation of a response from the true regression line.

RSE is considered a measure of lack of fit. So if it is large, the model is considered not to fit well whereas if RSE is small, the model may fit well.

$$ RSE = \sqrt{\dfrac{RSS}{n-2}}= \sqrt{\dfrac{\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}{n-2}} $$
Where n − 2 is the degrees of freedom.

Residual Standard Error

Goodness of fit is a measure of the strength of the linear relationship between two variables.  
The regression line $\hat{Y_t} = \hat{\alpha} + \hat{\beta X_t}$ obtained by the least square method is the best fitting line of all possible lines at least in the sense that the sum of squares of residuals is the smallest.

Goodness of Fit

If the null hypothesis $H_0$ is true, the statistical value has F distribution with degree of freedom 1 and (n-2) 
$$ F = \frac{SSEModel / 1}{SSE / (n-2)} = \frac{MSModel}{MSE} ~ F_{1, n-2}$$


F Statistics

Mean Square Error (MSE) reflects the degree of difference between the estimated value and the actual value. If the predicted response value is close to the real response value, MSE will be very small. If the predicted response value is substantially different from the real response value, MSE will be very large.
$$ MSE = \frac{1}{m} \sum_{i=1}^{m} (y_m - \hat{y_m})^2 $$


Mean Square Error (MSE) - Evaluation Model

R-Squared ($R^2$)—the proportion of variance explained—measures how well a prediction model for "regression" fits the given data. It is independent of the scale of $Y$.
$$0 \leq R^2 \leq 1$$
The larger the R-Squared value, the closer it is to 1, the better the regression model explains the variation of observations around its mean.
$$ R^2 = \frac{TSS-RSS}{TSS} = 1 - \frac{RSS}{TSS} = 1 - \frac{\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{n} (y_i - \bar{y})^2} $$

R-Squared (Coefficient of Determination)

The extent to which the linear regression model fits the data, the quality of linear regression fit, can be achieved by:
1. $ R^2 $
2. Residual Standard Error
3. Goodnesss of Fit
4. F Statistics
5. Mean Square Error (MSE)

University of Michigan - Ann Arbor

Linear regression is a linear approach to model a target prediction value based on independent variables.

Linear Regression

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Based on the number of independent variables used, there are two types of linear regression:
- Simple Linear Regression
- Multiple Linear Regression

Types of linear regression

Logistic regression is used to model discontinuous output choices - for example, when a variable is 'yes' or 'no' rather than a continuous value. In this case, logistic regression calculates the probability of a given categorical output, and decides on a threshold at which the model decides between categories to select for output. 

Logistic Regression

The terms of a linear model can be one of the following:
	
	- The Constant
	- A parameter multiplied by an independent variable (I.V)


Then, you build the equation by only adding the terms together. These rules limit the form to just one type:
Dependent variable = constant + parameter * IV + … + parameter * IV

Nonlinear regression models don’t fit this type.

Examples of nonlinear models include: 
	

When is a model considered linear?

This is a series of videos that introduces linear regression models and breaks down the process of linear regression.

[StatQuest with Josh Starmer]. (2020, February 21). Linear Regression and Linear Models [Video File]. Retrieved from [www.youtube.com/playlist?list=PLblh5JKOoLUIzaEkCLIUxQFjPIlapw8nU](https://www.youtube.com/playlist?list=PLblh5JKOoLUIzaEkCLIUxQFjPIlapw8nU) 

Linear Regression and Linear Models Videos

Here is an example of using R to do a simple linear regression. 
```R
#MASS package is a very large collection of data sets and #functions
library(MASS) 

#ISLR package includes the data sets associated with this #book
library(ISLR) 

lm.fit=lm(medv∼lstat,data=Boston) 
# medv as a response, lstat as a predictor

summary(lm.fit) # gives us pvalues and standard errors for the coeﬃcients, as well as the R2 statistic and F-statistic for the model

confint(lm.fit) # obtain a confidence interval

predict (lm.fit ,data.frame(lstat=c(5,10,15)), interval="confidence") #prediction in confidence intervals

predict (lm.fit ,data.frame(lstat=c(5,10,15)), interval="prediction") #prediction in prediction intervals

attach(Boston) # attach data to Boston

plot(lstat ,medv) 
abline(lm.fit ,lwd=3,col="red") 
# draw out the regression line
```
Expected output (plot and abline only):
![The expected output from the plot and abline commands](https://firebasestorage.googleapis.com/v0/b/onecademy-1.appspot.com/o/UploadedImages%2FWenfei-Tang_Wed%2C%2026%20Feb%202020%2020%3A51%3A58%20GMT.png?alt=media&token=5cf41866-c6fe-4732-88c5-2233fe889aa3) 

Example of Linear Regression Using R

We estimate the parameters $\beta_0, \beta_1, \beta_2, \dots, \beta_p$ such that we minimize the error in the following approximation:
$$Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots + \beta_p X_p + \epsilon$$

Training or Fitting a linear regression model

Assessing the accuracy of linear regression

Linear regression is an example of a parametric method because it assumes a linear functional form for f(X). But if the specified functional form is far from the truth, and prediction accuracy is our goal, then the parametric method will perform poorly. This leads to higher bias. There is also a well known statistical theory behind linear regression.

KNN is an example of non parametric method that uses feature similarity. It is pretty useful, because in the real world, most of the practical data does not obey the typical theoretical assumptions made (eg gaussian mixtures, linearly separable etc). Non-parametric methods do not explicitly assume a parametric form for f(X), and thereby provide an alternative and more flexible approach for performing regression. This means it will have less bias and is more robust with highly non-linear settings. However, KNN regression is not as widely studied as linear regression.

Comparison of Linear Regression with K-Nearest Neighbors

- Linearity of the response-predictor relations
- Normal distribution of residuals
-  Error terms, e1,e2,...e3, are uncorrelated
-  Error terms have a constant variance
-  Little or no Multicollinearity between the features
-  Homoscedasticity Assumption
-  Large sample size and few or no outliers

Assumptions of Linear Regression

The regression coefficient is $$\beta_1$$, in the regression equation $$y \approx \beta_0 + \beta_1 X$$, is an esitmate of the rate of change (slope) between X and Y.

Regression Coefficient

It is easier to implement, interpret, and infer from the estimation of linear regression. However, one can rarely find a perfect linear relationship in the real world. Estimation using nonlinear regression is more difficult to implement, interpret, and infer from, but it gives us more predictive power by relaxing the assumption of linearity of the model.

Linear regression vs. nonlinear regression

- L1 Regularization
- L2 Regularization
- Dropout Regularization
- Data Augmentation
- Early Stopping
- Tangent Distance
- Tangent Prop
- Manifold Tangent Classifier

Popular Regularization Techniques in Deep Learning

We can use  polynomial future transformation to transform a problem into a higher dimensional regression space. Adding these extra polynomial features allows us a much richer set of complex functions that we can use to fit to the data. This is like allowing polynomials to be fit to the training data instead of simply a straight line, but using the same least-squares criterion that minimizes mean squared error. This approach of adding new features like polynomial features is also very effective with classification.

Polynomial Future Transformation

To make up for the under-fitting or over-fitting problems of ordinary linear regression models, locally weighted linear regression introduce weights to loss function. 
Its loss is transmitted to:
$J(\theta)=\sum_{i=1}^m w^i (y^{(i)} - \theta^T x^{(i)})^2$
w is used to represent the weights. Its value is calculated using Gaussian kernel function:
$w^i = exp(\frac{(x-x^i)^2}{-2k^2})$

Learn Before

Related

Learn After