For the simple linear regression equation $$\hat{y}=\hat{\beta}_0+\hat{\beta}_1x$$, the Residual Sum of Squares (RSS) is minimized when the coefficient estimates are $$\hat{\beta}_1=\dfrac{\sum_{i=1}^n (x_i-\bar{x})(y_i-\bar{y})}{\sum_{i=1}^n (x_i-\bar{x})^2}$$ and $$\hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{x}$$, where $$\bar{x}$$ and $$\bar{y}$$ are the sample means.

University of Michigan - Ann Arbor

Google

The regression coefficient is $$\beta_1$$, in the regression equation $$y \approx \beta_0 + \beta_1 X$$, is an esitmate of the rate of change (slope) between X and Y.

Regression Coefficient

The Residual Sum of Squares (RSS) is the sum of the squared residuals (differences between predicted and actual values) across all data points. It is defined by the formula:

$$RSS = e_1^2 + e_2^2 + \\dots + e_n^2$$

Alternatively, in simple linear regression, it can be written as:

$$RSS = \\sum_{i=1}^n (y_i - \\hat{\\beta}_0 - \\hat{\\beta}_1 x_i)^2$$

where $$\\hat{\\beta}_0$$ and $$\\hat{\\beta}_1$$ are the estimated model parameters. A smaller RSS indicates that the regression model fits the data more closely.

Residual Sum of Squares (RSS)

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

A conventional measure of a treatment's effectiveness. It averages treatment efficacy over a population and can be found using a randomized control trial (RCT). It measures how much an increase in a single unit of treatment variable X will affect the outcome variable Y. When you have properly eliminated all confounding variables, you can obtain an estimate through running a regression.

Average Causal Effect (ACE)

When there is a confounder $$Z$$ for variables $$X$$ and $$Y$$, plotting all three variables forms a cloud of points in XYZ-space. The analogue of a regression line is a regression plane with the equation $$Y = aX + bZ + c$$. The coefficient $$a$$ is the regression coefficient of $$Y$$ on $$X$$ already adjusted for $$Z$$. It is called a partial regression coefficient and is written as $$r_{YX.Z}$$. If $$Z$$ is the only confounder, then $$a$$ represents the average causal effect of $$X$$ on $$Y$$.

Partial Regression Coefficient

Least Squares Coefficient Estimates

Residual Standard Error is an estimate of σ standard deviation. It assesses the quality of linear regression. It is the average deviation of a response from the true regression line.

RSE is considered a measure of lack of fit. So if it is large, the model is considered not to fit well whereas if RSE is small, the model may fit well.

$$ RSE = \sqrt{\dfrac{RSS}{n-2}}= \sqrt{\dfrac{\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}{n-2}} $$
Where n − 2 is the degrees of freedom.

Residual Standard Error

Mean Square Error (MSE) reflects the degree of difference between the estimated value and the actual value. If the predicted response value is close to the real response value, MSE will be very small. If the predicted response value is substantially different from the real response value, MSE will be very large.
$$ MSE = \frac{1}{m} \sum_{i=1}^{m} (y_m - \hat{y_m})^2 $$


Mean Square Error (MSE) - Evaluation Model

R-Squared ($R^2$)—the proportion of variance explained—measures how well a prediction model for "regression" fits the given data. It is independent of the scale of $Y$.
$$0 \leq R^2 \leq 1$$
The larger the R-Squared value, the closer it is to 1, the better the regression model explains the variation of observations around its mean.
$$ R^2 = \frac{TSS-RSS}{TSS} = 1 - \frac{RSS}{TSS} = 1 - \frac{\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{n} (y_i - \bar{y})^2} $$

Learn Before

Related