There may be some cases where score matching should be regularized, in which case we would denoise the score matching. We can do this by fitting a distribution $$p_{smoothed}(x) = \int p_{data}(y)q(x | y)dy$$ rather than the true $$p_{data}$$. Denoising score matching is useful when we don't have access to the true $$p_{data}$$, but rather only an empirical distribution defined by samples from it. This is one way of overcoming the partition function problem.

South Dakota State University

Google

Score matching helps to train a neural network model without evaluating the partition function , $Z$ or its derivative.
The strategy used is to minimize the expected squared difference between the derivatives of the model's log density with respect to the input and the derivatives of the data's log density with respect to the input.

$L(x,\theta) = \frac{1}{2} \| \nabla_x \log p_{model}(x;\theta),- \nabla_x \log p_{data}(x) \|_{2}^{2}$

$ J(\theta) = \frac{1}{2} E_{p_{data}}(x)L(x,\theta)$

$\theta^* = \min\limits_{\theta} J(\theta)$


Learn Before

Related