A supervised statistical learning model assumes: 1. A relationship between the matrix of input variables $$X$$ and the outcome $$Y$$. 2. Randomness and independence of $$\epsilon$$ from $$X$$. 3. $$\epsilon$$ should have a mean of 0.

University of Michigan - Ann Arbor

Google

The purpose of a statistical learning model is to use a statistical model and the training data to find the function $\hat{f}$ as an estimate of the unknown function $f$ that maps the matrix of input variables $X$ to the outcome vector $Y$, as best as possible.
$$Y = f(X) + \epsilon \approx \hat{f}(X)$$ 

Purpose of supervised statistical learning

In the supervised statistical learning model $$Y = f(X) + \epsilon$$, the term $$\epsilon$$ is an irreducible error that captures all variation in $$Y$$ that cannot be explained by $$f(X)$$. This includes unmeasurable factors, inherent randomness, and any other sources of variation in $$Y$$ that are independent of $$X$$.

The Error Term $$\epsilon$$ in Supervised Statistical Learning

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Supervised statistical learning assumptions

Most textbooks represent vectors (columns) of a tabular dataset $X$ as:
$$X_j=\begin{bmatrix}
    x_{1j} \\
    x_{2j} \\
    x_{3j} \\
    \vdots \\
    x_{nj} \\
\end{bmatrix}$$
So, the dataset can be represented as $$X=\begin{bmatrix}
    X_{1} & X_{2} & X_{3} & \dots  & X_{p}
\end{bmatrix}$$.

Formulating the dataset in supervised statistical learning

After choosing a functional form for $\hat{f}$, based on our domain knowledge or other techniques, we need to estimate the parameters of the model based on the training data. The process of finding optimal values for the parameters of $\hat{f}$ based on the training data is called "training" or "fitting" the model.

Training or Fitting a supervised statistical learning model

In supervised learning problems, a label, also known as a target, is a special attribute of an example that a model aims to predict. It represents the known outcome or ground truth that is not part of the model's input features.

Machine Learning Label

A prediction problem in statistical learning occurs when there are a significant number of observations and their corresponding outcomes, and the goal is to predict outcomes for a new set of observations where they are unavailable. Let $$X$$ be the set of all input variables and $$Y$$ be the set of corresponding outcomes. If $$\hat{f}$$ is the estimate of the function $$f$$ that maps $$X$$ to $$Y$$, and $$\hat{Y}$$ is the resulting prediction of $$Y$$, assuming the error term averages to zero ($$\bar{\epsilon}=0$$), the following holds:

$$Y=f(X)+\epsilon, \bar{\epsilon}=0 \implies \hat{Y}=\hat{f}(X)$$

Prediction in statistical learning

In supervised statistical learning models, the error term $$\epsilon$$ is assumed to be a random variable with a mean of 0.

Assumptions on $$\epsilon$$ in Supervised Statistical Learning

The error introduced when estimating a function $$f$$ with $$\hat{f}$$ is called reducible error because it can be reduced by using more suitable statistical models. Even if we could reduce the reducible error to zero, meaning $$\hat{Y}=f(X)$$, we would still suffer from irreducible error because $$Y=f(X)+\epsilon$$, which implies $$Y=\hat{Y}+\epsilon$$. Since the error term $$\epsilon$$ is independent of $$X$$, the irreducible error cannot be reduced regardless of the statistical techniques used. The expected squared error of predicting $$Y$$ using $$\hat{Y}$$ is decomposed as: $$E(Y-\hat{Y})^2 = E(f(X)+\epsilon-\hat{f}(X))^2 = \underbrace{[f(X)-\hat{f}(X)]^2}_{\text{Reducible}} + \underbrace{Var(\epsilon)}_{\text{Irreducible}}$$ Here, $$E(Y-\hat{Y})^2$$ is the expected value of the squared error, and $$Var(\epsilon)$$ is the variance of the error term $$\epsilon$$.

Learn Before

Related