NADE or a Neural Auto-regressive Density Estimator is largely identical to a standard Neural Auto-Regressive Network, but with one slight difference.

Input values $x_i$ are supplied into the network only after being weighted by weight vectors $W'_{j,k,i} = W_{k,i}$

Or the weight vector $W_{k,i}$ is reused for all inputs into hidden unit groups $h^{(j)}$ where $j>=i$

This allows for forward propagation of a NADE model to somewhat resemble the computation of missing inputs in a mean field inference for an RBM (Restricted Boltzmann Machine).

NADE

University of Toledo

The simplest form of auto-regressive network has no hidden units and no sharing
of parameters or features. And each conditional probability is parametrized as a linear
model.

Linear Auto-Regressive Network

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

If the variables are continuous, a linear auto-regressive model is merely another
way to formulate a multivariate Gaussian distribution.

A special degenerating condition of linear ARN

$O(d^2)$ parameters when there are d variables to model.

Magnitude of Parameters of Linear ARN

They may be trained with convex loss functions and sometimes admit closed form solutions

Advantages of Linear ARN

The model itself does not offer a way of increasing its capacity, so capacity must be raised using techniques like basis expansions of the input or the kernel trick.

Learn Before

Related

Learn After