- Divergence: The additional uncertainty induced by using probabilities from
one distribution to describe another distribution.

-Training sample: Data set to estimate the parameters. 

-Test Sample: Data set to address model accuracy measures.

-Scoring the right data- Similar to cross-validation using training and test data sets. 


University of Rochester

Accuracy: 

- Heuristic approach to the concept, we can have a high probability of making correct predictions and predict wrong most of the time yielding poor accuracy. 

-  Log scoring Rule for accuracy measure inherently uses the Bayes Theorem with joint probabilities (likelihood). 

Entropy: 

-  The uncertainty contained in a probability distribution. 

- Average log-prob of the event

-  Is closely related to its scientific meaning, the measure of disorder in a system, now in terms of probability. 



Entropy and Accuracy 

McElreath, R. (2020). Statistical Rethinking: A Bayesian Course with Examples in R and STAN (2nd ed.). Chapman and Hall/CRC.

Learn Before

Related