If the variables are continuous, a linear auto-regressive model is merely another
way to formulate a multivariate Gaussian distribution.

A special degenerating condition of linear ARN

They may be trained with convex loss functions and sometimes admit closed form solutions

Advantages of Linear ARN

The model itself does not offer a way of increasing its capacity, so capacity must be raised using techniques like basis expansions of the input or the kernel trick.

Disadvantages of Linear ARN

Neural Auto-Regressive Networks

A linear auto-regressive network has $$O(d^2)$$ parameters when modeling $$d$$ variables, because each of the $$d$$ conditional probabilities is parameterized as a linear model with no parameter sharing.

Number of Parameters in a Linear Auto-Regressive Network

The simplest form of auto-regressive network has no hidden units and no sharing
of parameters or features. And each conditional probability is parametrized as a linear
model.

University of Michigan - Ann Arbor

Auto-Regressive Networks are a kind of directed probability graph. The conditional probability is described using neural networks and the chain rule. The key definition of an Auto-Regressive Network is: $$p(x) =\prod_{i=1}^{n_i} p(x_i|x_{i-1},...,x_1)$$

Auto-Regressive Network

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

It introduces the relationship of Higher-order variables with fewer unique parameters and less computation.

Advantages of ARN

Linear Auto-Regressive Network

If an auto-regressive network has been trained on predicting values when given input in an arbitrary order, there is a simple technique that can be used to improve the accuracy of a prediction. Given $$n$$ input vectors, there are $$n!$$ many ways to arrange those vectors in order to predict the next output. Let $$n! \geq k$$ and let $$o^{(i)}$$ be a unique input permutation for $$i$$ ranging from 1 to $$k$$. $$p_{ensemble}(x) = \frac{1}{k}\sum_{i=1}^{k} p(x | o^{(i)})$$

Learn Before

Related

Learn After