Simple linear regression commonly uses a least squares approach, where the linear model attempts to minimize the residual distance between the actual points and the predicted line. Consequently, its residual is defined as:
$$e_i = y_i -\hat{y}_i$$

where $y_i$ represents the actual ith response value and $\hat{y}_i$ represents the ith response value of the linear model. 

Least Squares Approach

Estimates of $$\beta$$ from OLS are the best possible estimates that are unbiased and linear combinations of $$y$$, otherwise known as the Best Linear Unbiased Estimates (BLUE). 

Let $$\hat{\beta}$$ be unbiased (ie. $$E[\hat{\beta}] = \beta$$). Then $$Var(\hat{\beta_{OLS}}) \leq Var(\hat{\beta})$$.

Gauss-Markov Theorem (BLUE)

Linear regression uniquely presents an optimization problem with a direct analytic solution. By subsuming the bias term into the weight vector $$\mathbf{w}$$—which is done by appending a column of all $$1$$s to the design matrix $$\mathbf{X}$$—the training objective becomes minimizing $$\|\mathbf{y} - \mathbf{X}\mathbf{w}\|^2$$. Setting the derivative of this loss with respect to $$\mathbf{w}$$ equal to $$0$$ yields the intermediate derivative formula $$\partial_{\mathbf{w}} \|\mathbf{y} - \mathbf{X}\mathbf{w}\|^2 = 2 \mathbf{X}^	op (\mathbf{X} \mathbf{w} - \mathbf{y}) = 0$$, which directly gives the normal equation $$\mathbf{X}^	op \mathbf{y} = \mathbf{X}^	op \mathbf{X} \mathbf{w}$$. Solving for $$\mathbf{w}$$ provides the optimal parameters: $$\mathbf{w}^* = (\mathbf{X}^	op \mathbf{X})^{-1}\mathbf{X}^	op \mathbf{y}$$. This closed-form solution exists only if the design matrix has full rank, ensuring that the matrix $$\mathbf{X}^	op \mathbf{X}$$ is invertible.

Linear Regression Analytic Solution

Most common approach to fitting a linear model as close as possible to training data. Closeness in this case is minimizing the sum of squared errors. 

University of Michigan - Ann Arbor

We estimate the parameters $\beta_0, \beta_1, \beta_2, \dots, \beta_p$ such that we minimize the error in the following approximation:
$$Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \dots + \beta_p X_p + \epsilon$$

Training or Fitting a linear regression model

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

Learn Before

Related

Learn After