For the equation $\hat{y}=\bar{\beta}_0+\bar{\beta}_1x$, RSS is minimized (and the regression is "best") when 
$\hat{\beta}_1=\dfrac{\sum_{i=1}^n (x_i-\bar{x})(y_i-\bar{y})}{\sum_{i=1}^n (x_i-\bar{x})^2}$ 
and 
$\hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{x}$, 
where $\bar{x},\bar{y}$ are the sample means. 

University of Michigan - Ann Arbor

RSS is the aggregated squared residuals of all of the data. This is shown with the formula:

$RSS = e_1^2 + e_2^2 + ... + e_n^2$
or alternatively, 
$RSS=\sum_{i=1}^n (y_i-\hat{\beta}_0-\hat{\beta}_1x_i)^2$, 
where $\hat{\beta}_0, \hat{\beta}_1$ are the model parameters.

The RSS should be a smaller or lower value in any regression model. The smaller the RSS, the better the model fits the data


Residual Sum of Squares (RSS)

The regression coefficient is $$\beta_1$$, in the regression equation $$y \approx \beta_0 + \beta_1 X$$, is an esitmate of the rate of change (slope) between X and Y.

Regression Coefficient

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Residual Standard Error is an estimate of σ standard deviation. It assesses the quality of linear regression. It is the average deviation of a response from the true regression line.

RSE is considered a measure of lack of fit. So if it is large, the model is considered not to fit well whereas if RSE is small, the model may fit well.

$$ RSE = \sqrt{\dfrac{RSS}{n-2}}= \sqrt{\dfrac{\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}{n-2}} $$
Where n − 2 is the degrees of freedom.

Residual Standard Error

Mean Square Error (MSE) reflects the degree of difference between the estimated value and the actual value. If the predicted response value is close to the real response value, MSE will be very small. If the predicted response value is substantially different from the real response value, MSE will be very large.
$$ MSE = \frac{1}{m} \sum_{i=1}^{m} (y_m - \hat{y_m})^2 $$


Mean Square Error (MSE) - Evaluation Model

Least Squares Coefficient Estimates

R-Squared ($R^2$)—the proportion of variance explained—measures how well a prediction model for "regression" fits the given data. It is independent of the scale of $Y$.
$$0 \leq R^2 \leq 1$$
The larger the R-Squared value, the closer it is to 1, the better the regression model explains the variation of observations around its mean.
$$ R^2 = \frac{TSS-RSS}{TSS} = 1 - \frac{RSS}{TSS} = 1 - \frac{\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{n} (y_i - \bar{y})^2} $$

R-Squared (Coefficient of Determination)

A conventional measure of a treatment's effectiveness. It averages treatment efficacy over a population and can be found using a randomized control trial (RCT). It measures how much an increase in a single unit of treatment variable X will affect the outcome variable Y. When you have properly eliminated all confounding variables, you can obtain an estimate through running a regression.

Average Causal Effect (ACE)

When there is a condounder $Z$ for $X$ and $Y$, by plotting all three variables together, with each value of $(X, Y, Z)$ describing one point in space. In this case, the data will form a cloud of points in XYZ-space. The analogue of a regression line is a regression plane, which has an equation that looks like $Y = aX + bZ + c$. We can easily compute $a, b, c$ from the data.

The coefficient $a$ gives us the regression coefficient of $Y$ on $X$ already adjusted for $Z$. (It is called a partial regression coefficient and written $r_{YX.Z}$.)

If $Z$ is the only confounder, then a is the average causal effect of $X$ on $Y$.

Learn Before

Related