A Bayesian network's structure is made up of nodes (which can be multi-variate), edges between paired nodes, and a conditional probability distribution in each nodes. The nodes in a Bayesian network represent random variables and edges represent the conditional dependence of these nodes. We try to use this structure to model the conditional probability distribution of outcome variables (which are often causal) after observing new evidence.

How do Bayesian Networks work?

This article covers Bayesian networks and how they work by looking at different aspects of probabilistic graphical models.

[www.bayesserver.com/docs/introduction/bayesian-networks#_dynamic-bayesian-networks](https://www.bayesserver.com/docs/introduction/bayesian-networks#_dynamic-bayesian-networks) 

Helpful article about Bayesian Networks

Belief Propagation is an iterative message passing algorithm used within graphs or trees. In Bayesian Networks, bayesian likelihoods are calculated at certain nodes and passed up or down the network.

* If the message is passed downward (e.g. from parent to child), the child can update its beliefs using conditional probabilities.
* If the message is passed upward (e.g. from child to parent), the parent can update its beliefs by multiplying them by a likelihood ratio.

Belief Propagation in Bayesian Networks

Looking at related information based on the value of one variable is called conditioning on a variable. This is a helpful tool when programming a machine to update its beliefs: it can focus on relevant information while ignoring the rest.

Conditioning on a Variable

The conditional probability table displays conditional probabilities of mutually dependent random variables of a single variable contingent on the others.  Not to be confused with the Contingency Table, which describes the frequencies of data points in the given category. 

Conditional Probability Table

As mentioned before, Bayesian networks encode conditional independencies, i.e. we can use them to determine "independent" events.

Bayesian Networks and Independence

"Bayesian intelligence failed to bridge the gap between artificial and human intelligence. I'm sure you can figure out the missing ingredient: causality"



Criticism of Bayesian Network from the Judea Pearl

Most of the statistical methods are entirely objective, which means reasoning exclusively from data and experiments. But causal analysis and Bayesian analysis are different in that they involves some subjectivity.

Causal analysis requires some subjective commitments based on one's knowledge in the field before the causal digram is drawn and data is explored.

Similarly, Bayesian analysis combines objective evidence with the prior subjective belief.

From objectivity to subjectivity

Given that the guest chooses door 1 and the host opens door 3, we have: 


$P(1=car|3=opened) $

$= \frac{P(3=opened|1=car) *P(1=car)} {p(3=opened)} $

$=\frac{\frac{1}{2}*\frac{1}{3}}{p(3=opened)}$


P(3=opened)  is the probability that the host opens door 3 when guest picked door 1.

$P(3=opened)$ 
$= P(3=opened |1= car)*P(1= car)+P(3=opened|2= car)*P(2 = car)+P(3=opened |3 = car)*P(3 = car)$
$=\frac{1}{2}*\frac{1}{3}+1*\frac{1}{3}+0*\frac{1}{3}$
$=\frac{1}{2}$


$P(1=car|3=opened) $
$=\frac{\frac{1}{6}}{\frac{1}{2}}$
$=\frac{1}{3}$

Hence the possibility of car is behind door 1 is $1/3$, given guest selected door 1 and host opened door 3. 

Bayesian Solution for Monty Hall’s Paradox


- The COVID-19 testings tend to focus on already hospitalized people, not accounting for the people with mild symptoms and the asymptomatics leading to selection bias.  
- The differences in death rates across countries may not be attributed to clinical, demographic, and environmental factors.  
- Together with random sampling/testing, the Bayesian Network model allows causal explanations from data for better predictions.  


Summary of "COVID-19 infection and death rates: the need to incorporate causal explanations for the data and avoid bias in testing"

Structure Learning is a field of machine learning in which data is used to learn the links of a Bayesian network or Dynamic Bayesian network. The two major approaches for structure learning are the score-based approach and the constraint-based approach.

Structure Learning

Bayesian networks were developed to understand reasoning under conditions of uncertainty. It is based on Bayes Theorem and is a probabilistic model that represents a set of variables and their conditional dependencies by being able to find the probability of facts being true/false given already defined facts. A Bayesian network is represented by a directed acyclic graph which helps to visualize the probabilistic model for reviewing relationships between random variables and reasonings for causal probabilities given certain facts



University of Michigan - Ann Arbor

Bayes Theorem is about predicting the probability of an event A occurring given an event B has already occurred. We can use the inverse probability of this statment and the probability of each event to find this value. 
P(A) is called prior probability, P(B|A) is Likelihood, P(B) is evidence, and P(A|B) is posterior probability.


Bayes Theorem Overview

Created by Thomas Bayes, who posited that our estimate of the probability of an outcome was determined by two things:
- 1) prior probability of an outcome 
- 2) likelihood of an outcome

Bayesian Inference

Pearl, J., & Mackenzie, D. (2018). $\mathit{The \ Book \ of \ Why: \ The \ New \ Science \ of \ Cause \ and \ Effect.}$ Basic Books.

The Book of Why

A method of supervised classification that takes many features, assumed to be independent, and uses Bayes' Theorem to determine given those independent variables how likely the likelihood of another event occurring.

It's called "Naive" because it assumes that features are conditionally independent, given the class, i.e., for all instances of a given class, the features have little or no correlation with each other.

(Naive) Bayes Classifier

According to Bayes' theorem, we have
$$
P\left(Y=k | X=x\right)=\frac{P\left(Y=k\right) P\left(X=x | Y=k\right)}{\sum_{l=1}^{K} P\left(Y=l\right) P\left(X=x | Y=l\right)}\\
=\frac{\pi_{k} f_{k}(x)}{\sum_{l=1}^{K} \pi_{l} f_{l}(x)}.
$$
In LDA with the assumption of Gaussian distribution in every class, we have the Gaussian density 
$$
f_k(x)=\frac{1}{(2 \pi_k)^{p / 2}|\boldsymbol{\Sigma}|^{1 / 2}} \exp \left(-\frac{1}{2}(x-\mu_k)^{T} \boldsymbol{\Sigma}^{-1}(x-\mu_k)\right)
$$
for a $p$-dimensional random vector $X$ with the distribution $N(\mu_k, \mathbf{\Sigma})$. All we need to do is to plug $f_k(x)$ into Bayes' theorem above and find out the $k$ which maximize $P\left(Y=k | X=x\right)$. LDA is a Bayes classifier with the assumption of Gaussian distribution in every class.

Using Bayes' theorem in LDA

Bayesian Networks

Example 1: Imagine a billiard table, and suppose that the billiard track bounces off the table many times, so we don't know where the ball will end up. Let's say the billiard table is L, and the probability that the ball ends up x feet from the edge is x/L.

The inverse probability problem: Suppose we observe that the ball's final stop is a foot away from the table, but we don't know the length of the table, L.

Bayes' main work is to break this cognitive asymmetry and propose a method to estimate the inverse probability.


Inverse-probability: break cognitive asymmetry 

This example in the book is a discussion on how we can update our beliefs using Bayes's rule.  The question the example tries to address is, "What is the probability that she has breast cancer, given that the test came out positive?", which can be expressed as P(D | T), where D is the hypothesis, disease, and T is the evidence, test.

$$
        $$(Updated probability of D) = P(D | T) = (likelihood ratio)) × (prior probability of D)$$
      $$

(Note) The "likelihood ratio" is given by P(T | D)/P(T). 



Mammogram (Breast Cancer Screening Example)

There are three out of four binary classification metrics mentioned in the Book of Why; true positive, false positive, and false negative.  The fourth one that is not mentioned in the book is true negative.  The following are the two primary building blocks of such four metrics: 

1. True vs. False: These are about prediction's correctness whether a prediction outcome was made accurately/ correctly.  The correct prediction is True, and the incorrect prediction is False.
2. Positive vs. Negative: These are about a binary classification -(e.g.) In the mammogram example in the book, "positive" is positive test results (breast cancer), whereas "negative" is negative test results (not breast cancer).  

The combinations of the above concepts result in True positive, True negative, false positive, and false negative, each of which is explained in the subsequent nodes to follow.  


Binary Classification Metrics

Inverse probability, as supposed to forward probability, is a conditional probability that a hypothesis (cause), H, is true given that the evidence (effect), E is true.  Inverse probabiity is harder to estimate than estimating a forward probability.  

It can be expressed as:

	 $$ P(H | E) $$


Inverse Probability

Forward probability, as supposed to inverse probability, is the conditional probability that an evidence (effect), E, is true given that the hypothesis (cause), H is true.  For the definition of inverse probability, see the node, "Inverse Probability Defined." 

It can be expressed as:

  $$P(E | H)$$

Estimating the forward probability is easier than estimating the inverse probability because, as long as we know the cause, it is easy to estimate the probability of the effect.  

Pearl calls forward probability, calculating conditional probability in the direction, in which "our judgment is more reliable."


Forward Probability

Bayes' Theorem can also be used as the following :

$$ P(A | B) = \frac{P(A \cap B)}{P(B)}$$
The way to think about this is that given two events A and B, the probability that A happens, given B occurs, is the probability of their intersection occuring i.e. both events occuring, over the probability that B occurs.
The easiest way to see why this works is for Independent events. If A and B are independent, the probability that they both occur is the product of their individual probabilities. Which then simplifies the above equation to :
$$ P(A|B)    =    \frac{P(A \cap B)}{P(B)}$$
$$ \implies P(A|B) = \frac{P(A).P(B)}{P(B)}$$
$$ \implies P(A|B) = P(A)$$
Which makes sense since the events are independent.

Alternate Bayes' Theorem Equation

The likelihood of an outcome represented by the extent to which available evidence points to the outcome.

Likelihood (Bayesian Inference)

Commonly referred to as the prior, it is the prior probability of an outcome occurring, represented by our initial belief about the probability of an outcome.

Prior Probability (Bayesian Inference)

Updating your beliefs can be boring or uncomfortable but it very beneficial in the long run. Look for subtle signals in noise and avoid wishful thinking. The world doesn't work the way we want it so ideally we would behave like rational Bayesian updaters.

Learn Before

Related

Learn After