When looking at the entropy of a given system, the maximum entropy tells us what (according to the law of large numbers ) we should expect the set of disorder to converge to. We can see the tendency of the is convergence described in the Gaussian distribution where even though certain events are just as likely on paper, we can see in the aggregate where the most frequently occurring events take place. It is also good to remember the rule of thumb looking at a gaussian distribution describing the entropy of a system that the lower the maximum of the gaussian curve is: the higher the entropy of the system as that means the more equally likely a given event is of occurring. The overall lesson one should take from this is that if a given collection of measurements has finite variance, then the Gaussian distribution represents the most conservative probability distribution to assign those measurements. 

Information Entropy

Similiar to what we have seen in the Gaussian distribution's relationship to entropy, binomials with their binary outcomes and constant probabilities result in the generation of an even/conservative distribution. We can also apply binomial distributions when we assume that a constant expected value.

Binomial application in entropy

We seek a measure of uncertainty that meets the following criteria:
1)  the measure should be continuous
2) it should increase as the number of possible events increases
3)  it should be additive

Syracuse University

This textbook will primarily be used for subsequent nodes about Bayesian Statistics. It provides a general overview of what Bayesian Statistics is, including example problems and codes. 

Learn Before

Related

Learn After