Estimates of $$\beta$$ from OLS are the best possible estimates that are unbiased and linear combinations of $$y$$, otherwise known as the Best Linear Unbiased Estimates (BLUE). 

Let $$\hat{\beta}$$ be unbiased (ie. $$E[\hat{\beta}] = \beta$$). Then $$Var(\hat{\beta_{OLS}}) \leq Var(\hat{\beta})$$.

University of Michigan - Ann Arbor

Most common approach to fitting a linear model as close as possible to training data. Closeness in this case is minimizing the sum of squared errors. 

Ordinary Least Squares (OLS)

Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: data mining, inference, and prediction. Springer Science & Business Media.

The Elements of Statistical Learning: Data Mining, Inference and Prediction

Simple linear regression commonly uses a least squares approach, where the linear model attempts to minimize the residual distance between the actual points and the predicted line. Consequently, its residual is defined as:
$$e_i = y_i -\hat{y}_i$$

where $y_i$ represents the actual ith response value and $\hat{y}_i$ represents the ith response value of the linear model. 

Least Squares Approach

Gauss-Markov Theorem (BLUE)

Linear regression uniquely presents an optimization problem with a direct analytic solution. By subsuming the bias term into the weight vector $$\mathbf{w}$$—which is done by appending a column of all $$1$$s to the design matrix $$\mathbf{X}$$—the training objective becomes minimizing $$\|\mathbf{y} - \mathbf{X}\mathbf{w}\|^2$$. Setting the derivative of this loss with respect to $$\mathbf{w}$$ equal to $$0$$ yields the intermediate derivative formula $$\partial_{\mathbf{w}} \|\mathbf{y} - \mathbf{X}\mathbf{w}\|^2 = 2 \mathbf{X}^	op (\mathbf{X} \mathbf{w} - \mathbf{y}) = 0$$, which directly gives the normal equation $$\mathbf{X}^	op \mathbf{y} = \mathbf{X}^	op \mathbf{X} \mathbf{w}$$. Solving for $$\mathbf{w}$$ provides the optimal parameters: $$\mathbf{w}^* = (\mathbf{X}^	op \mathbf{X})^{-1}\mathbf{X}^	op \mathbf{y}$$. This closed-form solution exists only if the design matrix has full rank, ensuring that the matrix $$\mathbf{X}^	op \mathbf{X}$$ is invertible.

Learn Before

Related