Learn Before
Information Entropy
When looking at the entropy of a given system, the maximum entropy tells us what (according to the law of large numbers ) we should expect the set of disorder to converge to. We can see the tendency of the is convergence described in the Gaussian distribution where even though certain events are just as likely on paper, we can see in the aggregate where the most frequently occurring events take place. It is also good to remember the rule of thumb looking at a gaussian distribution describing the entropy of a system that the lower the maximum of the gaussian curve is: the higher the entropy of the system as that means the more equally likely a given event is of occurring. The overall lesson one should take from this is that if a given collection of measurements has finite variance, then the Gaussian distribution represents the most conservative probability distribution to assign those measurements.
0
1
Tags
Bayesian Statistics
Statistics
Data Science