A Neural Autoregressive Density Estimator (NADE) is largely identical to a standard Neural Auto-Regressive Network, but with one key difference in parameter sharing. Input values $$x_i$$ are supplied into the network only after being weighted by weight vectors $$W'_{j,k,i} = W_{k,i}$$. In other words, the weight vector $$W_{k,i}$$ is reused for all inputs into hidden unit groups $$h^{(j)}$$ where $$j \geq i$$. This allows the forward propagation of a NADE model to resemble the computation of missing inputs in a mean field inference for a Restricted Boltzmann Machine (RBM).

Neural Autoregressive Density Estimator (NADE)

University of Toledo

The simplest form of auto-regressive network has no hidden units and no sharing
of parameters or features. And each conditional probability is parametrized as a linear
model.

Linear Auto-Regressive Network

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

If the variables are continuous, a linear auto-regressive model is merely another
way to formulate a multivariate Gaussian distribution.

A special degenerating condition of linear ARN

They may be trained with convex loss functions and sometimes admit closed form solutions

Advantages of Linear ARN

The model itself does not offer a way of increasing its capacity, so capacity must be raised using techniques like basis expansions of the input or the kernel trick.

Disadvantages of Linear ARN

Neural Auto-Regressive Networks

A linear auto-regressive network has $$O(d^2)$$ parameters when modeling $$d$$ variables, because each of the $$d$$ conditional probabilities is parameterized as a linear model with no parameter sharing.

Learn Before

Related

Learn After