A function of a function $f$ is called a **functional** $J[f]$.

Functionals

The calculus of variations is a mathematical tool that helps with solving an optimization problem with respect to a function. 

University of Connecticut

South Dakota State University

When we want to learn only one conditional statistic of y given x, instead of learning a full probability distribution p(y | x;θ), we can use a cost functional instead of just a cost function. In this way, we can view learning as choosing a function instead of choosing a set of parameters and our cost functional can be designed to have its minimum occur at a desired specific function. 

Learning Conditional Statistics

Inference means understanding the association of changes in the outcome $Y$ and changes in the predictors matrix $X$, i.e., estimating the characteristics of a function $f$ that maps the values of $X$ to corresponding values of $Y$.

Inference in statistical learning

A functional is a mapping from functions to real numbers.

Functional

Calculus of variations

The mean absolute error is one type of cost function that predicts the median value of y for each x, as long as such a function may be described by the family of functions that are optimized over. Unfortunately, mean absolute error may often end up with poor results when used with gradient-based optimization.

Mean absolute error

In both prediction and inference, we estimate a function $f$ that maps the values of predictors $X$ to outcome $Y$.
In prediction, our goal is to estimate the function $f$ based on the available values of $X$ and corresponding $Y$ so that we can use $f$ to predict the values of $Y$ for the values of $X$ for which we don't know the corresponding values of $Y$. So, the function $\hat{f}$ can be treated as a black box, and our main objectives are to improve the accuracy and precision of the prediction.
In inference, we already know all the corresponding values of $X$ and $Y$, and we're interested in understanding the characteristics of the function $f$ that maps the values of $X$ to their corresponding values of $Y$. So, $\hat{f}$ cannot be treated as a black box and our main objective is to find the exact functional form of $\hat{f}$.

Inference vs. prediction in statistical learning

1. Is the relation between the predictors $X_1, X_2, X_3, ..., X_P$ and the outcome $Y$ causal?
2. Is the correlation between the predictors $X_1, X_2, X_3, ..., X_P$ and the outcome $Y$ linear or non-linear?
3. Which predictors $X_1, X_2, X_3, ..., X_P$ are correlated, and which ones are not correlated with outcome $Y$?
4. What is the direction of the correlation between each predictor $X_1, X_2, X_3, ..., X_P$ and the outcome $Y$?
5. What is the magnitude of the correlation between each predictor $X_1, X_2, X_3, ..., X_P$ and the outcome $Y$?

Types of inferential questions in statistical learning

The inference engine is a machine that accepts Assumptions, Queries, and Data, and produces Testable implications, Estimand, and Estimate. The implications and estimand do not necessarily need to be presented at the end, but they are both implicit within the model. The first of the outputs is a Y/N decision as to whether the given query could be answered under the existing causal model, assuming perfect data. If the answer is Yes, the inference engine next produces an Estimand, a mathematical formula that generates the answer from any hypothetical data. If the answer is No, then the engine returns to the assumptions stage to refine the model, through either simplifying assumptions or adding new scientific knowledge. Finally, after the inference engine has received the Data, it will use the Estimand to produce an actual Estimate for the answer, along with the amount of uncertainty in that estimate.

Inference Engine

Exact inference can be described as an optimization problem. Thus, approximate inference algorithms can be derived by approximating the underlying optimization. We assume a probabilistic model consisting of observed variables $v$ and latent variables $h$. Sometimes it is too difficult to compute the log-probability ($log p(v;\theta)$) of the observed data, so we can instead compute a lower bound $\mathcal{L}(v, \theta,q)$ on $log p(v;\theta)$, called the evidence lower bound (ELBO), or the negative variational free energy. 

Inference as an Optimization Problem

Approximate Inference affects the learning process which affects the accuracy of the algorithm. A behavior is seen in which approximating assumptions become self-fulfilling. Training the model with a unimodal approximate posterior will result in a posterior that is closer than would result from training with exact inference. Computing the harm by variational approximation is very difficult.

Inference Effects on Learning

Linear models are a good choice because they allow us to easily understand the relationship between X1, X2, X3.... and Y. These types of linear models are more restrictive and therefore easier to interpret. However, they do not allow us to consider statistical situations in which nonlinear relationships exist. 
Reference: James, G., Witten, D., Hastie, T., & Tibshirani, R. (2017). An introduction to statistical learning: with applications in R. New York: Springer.


Learn Before

Related

Learn After