Learn Before
The Partition Function (introduction).
The partition function is an integral or sum over unnormalized probabilities. This function, often denoted by Z, is used to normalize an unnormalized distribution into a valid distribution.
where Z is:
or
Z = partition function.
= Unnormalized distribution.
= Normalized distribution.
In the context of deep learning, Z is usually intractable, meaning that it often needs to be approximated.
0
2
Contributors are:
Who are from:
Tags
Data Science
Foundations of Large Language Models Course
Computing Sciences
Learn After
Pseudolikelihood
Log-Likelihood Gradient
Log-Likelihood Gradient
Normalizing Model Outputs
A model produces unnormalized scores for three possible outcomes: {Outcome A: 8, Outcome B: 10, Outcome C: 2}. To convert these scores into a valid probability distribution, a normalization constant must be calculated by summing all the unnormalized scores. What is the final, normalized probability for Outcome B?
Computational Cost of Normalization