RSS is the aggregated squared residuals of all of the data. This is shown with the formula:

$RSS = e_1^2 + e_2^2 + ... + e_n^2$
or alternatively, 
$RSS=\sum_{i=1}^n (y_i-\hat{\beta}_0-\hat{\beta}_1x_i)^2$, 
where $\hat{\beta}_0, \hat{\beta}_1$ are the model parameters.

The RSS should be a smaller or lower value in any regression model. The smaller the RSS, the better the model fits the data


Residual Sum of Squares (RSS)

A data scientist is building a simple linear model for a set of data points. Four potential lines are generated to fit the data. According to the least squares approach, which of the following lines would be considered the best fit for the data?

A simple linear model is defined by the equation $\hat{y} = 1.5x + 4$. For an observed data point where the actual response value ($y$) is 12 when the predictor value ($x$) is 5, what is the residual?

Calculating a Residual

Evaluate the proposed linear model's fit to the data provided in the case study. In your evaluation, calculate the difference between each actual response value and the value predicted by the model. Based on these differences, is this model a good representation of the data according to the principle of finding a line that is as close as possible to all data points? Justify your answer.

Evaluating a Linear Model's Fit

Simple linear regression commonly uses a least squares approach, where the linear model attempts to minimize the residual distance between the actual points and the predicted line. Consequently, its residual is defined as:
$$e_i = y_i -\hat{y}_i$$

where $y_i$ represents the actual ith response value and $\hat{y}_i$ represents the ith response value of the linear model. 

University of Michigan - Ann Arbor

Google

Simple Linear Regression predicts a model for the relationship between a predictor matrix (independent variable) $X$ and the response (dependent variable) $Y$.
$$Y=\beta _1 X+\beta _0+\epsilon$$
The coefficients $\beta _1$ and $\beta _0$ are called slope and intercept, respectively. The estimated values for these coefficients are shown as $\hat{\beta _1}$ and $\hat{\beta _0}$.
$$\hat{Y}=\hat{\beta _1} X+\hat{\beta _0}$$

Simple Linear Regression

Most common approach to fitting a linear model as close as possible to training data. Closeness in this case is minimizing the sum of squared errors. 

Ordinary Least Squares (OLS)

In predictive modeling, ŷ (pronounced 'y-hat') represents the predicted or estimated value for a dependent variable, y. It is the output generated by a model based on a given set of input features. The primary objective during model training is to adjust the model's parameters so that its predictions, ŷ, are as close as possible to the actual, observed values of y.

Predicted Value (ŷ)

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Reference of Foundations of Large Language Models Course

Least Squares Approach

```python
#imports 
import numpy as np 
from sklearn.linear_model import LinearRegression

#make up data 
array_x = np.array([5, 10, 15, 20, 25]).reshape((-1, 1))
#calling reshape makes sure that the array is 2D 
array_y = np.array([5, 3, 15, 23, 6])

#create a model 
model_xy = LinearRegression()

#using model 
mode.fit(array_x, array_y)

#getting results 
rsq_val = model_xy(array_x, array_y)
intercept = model_xy.intercept_
slope = model_xy.coef_

#predicting a response 
y_pred = model_xy.predict(x)

```

Python: Simple Linear Regression 

In R, lm() is a function that can take many parameters. Common parameters include formula, which describes a model, and data, which contains a dataset. 
 

In simple linear regression in R, a formula parameter is written YVAR~XVAR, where XVAR is the predictor variable and YVAR is the response variable. 

A simple linear regression equation with data and formula parameters may look like:
``` R
lm(formula = YVAR~XVAR, data = datavar)
```


Simple Linear Regression in R: lm()

Link: $\href{http://www.sthda.com/english/articles/40-regression-analysis/165-linear-regression-essentials-in-r/}{sthda}$
The image references a simple linear regression line, where the 'e' represents the residual error, 'b0' represents the y-intercept, and 'b1' represents the change in Y (response value) for one unit increase in X.

Image Reference: Simple Linear Regression

Estimates of $$\beta$$ from OLS are the best possible estimates that are unbiased and linear combinations of $$y$$, otherwise known as the Best Linear Unbiased Estimates (BLUE). 

Let $$\hat{\beta}$$ be unbiased (ie. $$E[\hat{\beta}] = \beta$$). Then $$Var(\hat{\beta_{OLS}}) \leq Var(\hat{\beta})$$.

Gauss-Markov Theorem (BLUE)

Linear regression uniquely presents an optimization problem with a direct analytic solution. By subsuming the bias term into the weight vector $$\mathbf{w}$$—which is done by appending a column of all $$1$$s to the design matrix $$\mathbf{X}$$—the training objective becomes minimizing $$\|\mathbf{y} - \mathbf{X}\mathbf{w}\|^2$$. Setting the derivative of this loss with respect to $$\mathbf{w}$$ equal to $$0$$ yields the intermediate derivative formula $$\partial_{\mathbf{w}} \|\mathbf{y} - \mathbf{X}\mathbf{w}\|^2 = 2 \mathbf{X}^	op (\mathbf{X} \mathbf{w} - \mathbf{y}) = 0$$, which directly gives the normal equation $$\mathbf{X}^	op \mathbf{y} = \mathbf{X}^	op \mathbf{X} \mathbf{w}$$. Solving for $$\mathbf{w}$$ provides the optimal parameters: $$\mathbf{w}^* = (\mathbf{X}^	op \mathbf{X})^{-1}\mathbf{X}^	op \mathbf{y}$$. This closed-form solution exists only if the design matrix has full rank, ensuring that the matrix $$\mathbf{X}^	op \mathbf{X}$$ is invertible.

Linear Regression Analytic Solution

In machine learning, an objective function quantifies the goal of an algorithm, which is typically to minimize an error measure (a cost function) or maximize a probability function. For supervised learning problems like classification and prediction, an error function is defined to capture the discrepancy between the known correct outputs and the model's predicted outputs. The learning process then aims to find model parameters that minimize this function.

Objective Function in Machine Learning

The predicted value, denoted as $\hat{y}$, is formally determined as the argument that optimizes a specific objective function. This is expressed in the general form `$\hat{y}$ = arg ...`, where `arg` typically represents either `argmin` (argument of the minimum) or `argmax` (argument of the maximum). The choice between `argmin` and `argmax` depends on whether the objective is to minimize a loss function or maximize a likelihood function, respectively.

Formal Definition of the Predicted Value (ŷ)

A real estate company uses a machine learning model to estimate the market value of houses. For a specific house with 3 bedrooms and 2,000 square feet of living space, the model calculates an estimated value of $450,000. The house later sells for an actual price of $465,000. In the context of this predictive model, what does the $450,000 figure represent?

A data scientist builds a model to predict a student's final exam score based on their hours spent studying. For one student who studied for 10 hours, the model predicts a score of 85. The student's actual score on the exam was 88. Identify the predicted value and the actual value in this scenario, and explain what the difference between these two values indicates about the model's performance for this specific prediction.

Analyzing Model Predictions

The expression $\text{Pr}^t(y|c, z)$ denotes the conditional probability of an output variable $y$ given two conditioning variables, $c$ and $z$. The superscript $t$ can indicate a specific state or version of the model, such as at a particular training step. This formulation is central to probabilistic predictive models, where it is used to quantify the likelihood of a particular outcome based on given context and other factors.

Conditional Probability Pr^t(y|c, z)

A predictive model was trained to estimate a student's final exam score based on their hours spent studying. Analyze the table below, which shows the actual scores and the model's predicted scores for four different students. Based on this data, describe the model's performance, paying close attention to any patterns in its errors.

Learn Before

Related

Learn After