In information theory, the cross-entropy from a true probability distribution $$P$$ to a subjective probability distribution $$Q$$, denoted as $$H(P, Q)$$, is defined as the expected surprisal of an observer who uses the subjective probabilities $$Q$$ when observing data that was actually generated according to the true probabilities $$P$$. Mathematically, this is expressed as:

$$H(P, Q) \stackrel{	extrm{def}}{=} \sum_j - P(j) \log Q(j)$$

Cross-Entropy in Information Theory

Surprisal quantifies the level of surprise experienced by an observer upon witnessing an event $$j$$ to which they had assigned a subjective probability $$P(j)$$. It is defined mathematically as:
$$-\log P(j)$$
Events with lower probabilities yield higher surprisal, reflecting that they convey more information when they occur.

Claude

Probability is the branch of math concerned with the likelihood of an event occuring, or the likelihood that an event is true. 

Probability Theory

Dive into Deep Learning

- https://en.wikipedia.org/wiki/Probability#:~:text=Probability%20is%20the%20branch%20of,event%20and%201%20indicates%20certainty.
- Morin, D. (2016). Probability for the Enthusiastic Beginner. Las Vegas, Nevada: CreateSpace Independent Publishing Platform.
- https://en.wikipedia.org/wiki/Permutation

Probability References

- Combinatorics
- Probability Distributions
- Random Variables
- Independence
- Chain Rules

Topics in Probability Theory

This example demonstrates how to calculate the probability of a specific outcome from multiple independent events. Consider a scenario where three friends each flip a fair coin once, and the question is: what is the probability that exactly one of them flips heads? 

To solve this, we first identify the sample space, which consists of all possible outcomes. Since each of the three flips has two outcomes (Heads or Tails), there are $2^3 = 8$ total possibilities: HHH, HHT, HTH, HTT, THH, THT, TTH, TTT. 

Next, we identify the favorable outcomes that match the condition of having exactly one head: HTT, THT, and TTH. There are 3 such outcomes. 

The probability is the ratio of favorable outcomes to the total number of outcomes, which is 3/8.

Calculating the Probability of Exactly One Head in Three Coin Flips

The binomial probability formula is used to determine the probability of obtaining exactly *k* successes in a sequence of *n* independent trials. The formula is expressed as $\binom{n}{k} p^k (1-p)^{n-k}$, where *p* represents the probability of success on a single trial, and the binomial coefficient $\binom{n}{k}$ calculates the number of ways to choose *k* successes from *n* trials.

Binomial Probability Formula

A person rolls two standard, fair six-sided dice at the same time. What is the probability that the sum of the numbers shown on the top faces is exactly 5?

Based on your understanding of probability, analyze your friend's claim and conclusion. Is this single experiment of 10 flips sufficient evidence to conclude the coin is not a standard, fair coin? Explain your reasoning.

Evaluating a Claim about a 'Lucky' Coin

A fair spinner is divided into four equal sections, colored Red, Blue, Green, and Yellow. If the spinner is spun three independent times, what is the probability that it lands on 'Red' exactly twice? Explain your reasoning by identifying the total number of possible outcomes and the number of favorable outcomes.

Learn Before

Related

Learn After