In both prediction and inference, we estimate a function $f$ that maps the values of predictors $X$ to outcome $Y$.
In prediction, our goal is to estimate the function $f$ based on the available values of $X$ and corresponding $Y$ so that we can use $f$ to predict the values of $Y$ for the values of $X$ for which we don't know the corresponding values of $Y$. So, the function $\hat{f}$ can be treated as a black box, and our main objectives are to improve the accuracy and precision of the prediction.
In inference, we already know all the corresponding values of $X$ and $Y$, and we're interested in understanding the characteristics of the function $f$ that maps the values of $X$ to their corresponding values of $Y$. So, $\hat{f}$ cannot be treated as a black box and our main objective is to find the exact functional form of $\hat{f}$.

Inference vs. prediction in statistical learning

1. Is the relation between the predictors $X_1, X_2, X_3, ..., X_P$ and the outcome $Y$ causal?
2. Is the correlation between the predictors $X_1, X_2, X_3, ..., X_P$ and the outcome $Y$ linear or non-linear?
3. Which predictors $X_1, X_2, X_3, ..., X_P$ are correlated, and which ones are not correlated with outcome $Y$?
4. What is the direction of the correlation between each predictor $X_1, X_2, X_3, ..., X_P$ and the outcome $Y$?
5. What is the magnitude of the correlation between each predictor $X_1, X_2, X_3, ..., X_P$ and the outcome $Y$?

Types of inferential questions in statistical learning

The inference engine is a machine that accepts Assumptions, Queries, and Data, and produces Testable implications, Estimand, and Estimate. The implications and estimand do not necessarily need to be presented at the end, but they are both implicit within the model. The first of the outputs is a Y/N decision as to whether the given query could be answered under the existing causal model, assuming perfect data. If the answer is Yes, the inference engine next produces an Estimand, a mathematical formula that generates the answer from any hypothetical data. If the answer is No, then the engine returns to the assumptions stage to refine the model, through either simplifying assumptions or adding new scientific knowledge. Finally, after the inference engine has received the Data, it will use the Estimand to produce an actual Estimate for the answer, along with the amount of uncertainty in that estimate.

Inference Engine

The calculus of variations is a mathematical tool that helps with solving an optimization problem with respect to a function. 

Calculus of variations

Exact inference can be described as an optimization problem. Thus, approximate inference algorithms can be derived by approximating the underlying optimization. We assume a probabilistic model consisting of observed variables $v$ and latent variables $h$. Sometimes it is too difficult to compute the log-probability ($log p(v;\theta)$) of the observed data, so we can instead compute a lower bound $\mathcal{L}(v, \theta,q)$ on $log p(v;\theta)$, called the evidence lower bound (ELBO), or the negative variational free energy. 

Inference as an Optimization Problem

Approximate Inference affects the learning process which affects the accuracy of the algorithm. A behavior is seen in which approximating assumptions become self-fulfilling. Training the model with a unimodal approximate posterior will result in a posterior that is closer than would result from training with exact inference. Computing the harm by variational approximation is very difficult.

Inference Effects on Learning

Linear models are a good choice because they allow us to easily understand the relationship between X1, X2, X3.... and Y. These types of linear models are more restrictive and therefore easier to interpret. However, they do not allow us to consider statistical situations in which nonlinear relationships exist. 
Reference: James, G., Witten, D., Hastie, T., & Tibshirani, R. (2017). An introduction to statistical learning: with applications in R. New York: Springer.


Linear Models in Statistical Inference 

Inference means understanding the association of changes in the outcome $Y$ and changes in the predictors matrix $X$, i.e., estimating the characteristics of a function $f$ that maps the values of $X$ to corresponding values of $Y$.

University of Michigan - Ann Arbor

Use cases of supervised statistical learning

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Prediction problem is defined when we have a significant number of observations and their corresponding outcomes, and we'd like to predict the outcomes for another set of observations where the corresponding outcomes are unavailable. We call the set of all input variables $X$, and the corresponding set of all outcomes $Y$. If we define $\hat{f}$ as the estimate of the function $f$ that we need to estimate to map $X$ to $Y$, and we define $\hat{Y}$ as the resulting prediction of $Y$, and considering the fact that the error term averages to 0, the following holds:
$$Y=f(X)+\epsilon, \bar{\epsilon}=0, \implies \hat{Y}=\hat{f}(X)$$

Learn Before

Related

Learn After