In supervised statistical learning models, the error term $$\epsilon$$ is assumed to be a random variable with a mean of 0.

University of Michigan - Ann Arbor

Google

In the supervised statistical learning model $$Y = f(X) + \epsilon$$, the term $$\epsilon$$ is an irreducible error that captures all variation in $$Y$$ that cannot be explained by $$f(X)$$. This includes unmeasurable factors, inherent randomness, and any other sources of variation in $$Y$$ that are independent of $$X$$.

The Error Term $$\epsilon$$ in Supervised Statistical Learning

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

A supervised statistical learning model assumes: 1. A relationship between the matrix of input variables $$X$$ and the outcome $$Y$$. 2. Randomness and independence of $$\epsilon$$ from $$X$$. 3. $$\epsilon$$ should have a mean of 0.

Supervised statistical learning assumptions

Assumptions on $$\epsilon$$ in Supervised Statistical Learning

The error introduced when estimating a function $$f$$ with $$\hat{f}$$ is called reducible error because it can be reduced by using more suitable statistical models. Even if we could reduce the reducible error to zero, meaning $$\hat{Y}=f(X)$$, we would still suffer from irreducible error because $$Y=f(X)+\epsilon$$, which implies $$Y=\hat{Y}+\epsilon$$. Since the error term $$\epsilon$$ is independent of $$X$$, the irreducible error cannot be reduced regardless of the statistical techniques used. The expected squared error of predicting $$Y$$ using $$\hat{Y}$$ is decomposed as: $$E(Y-\hat{Y})^2 = E(f(X)+\epsilon-\hat{f}(X))^2 = \underbrace{[f(X)-\hat{f}(X)]^2}_{\text{Reducible}} + \underbrace{Var(\epsilon)}_{\text{Irreducible}}$$ Here, $$E(Y-\hat{Y})^2$$ is the expected value of the squared error, and $$Var(\epsilon)$$ is the variance of the error term $$\epsilon$$.

Learn Before

Related