Probability Measures and Bayes' Rule

Week 2

Reading

FT-2 | Probability Measure

We have already considered the measurable space $(\mathrm{\Omega },\mathrm{ℱ︀})$ of the sample space and $𝜎$-algebra. We can now define the probability measure $\mathbb{P}$ as a measure defined on events in the $𝜎$-algebra $\mathrm{ℱ︀}$, taking values in the unit interval $[0,1]$. This measures the size of the events, which has two properties: the probability of $\mathrm{\Omega }$ is 1, which means that all outcomes are accounted for in the sample space, and that the measure is countably additive, which allows us to extend the probability of individual events to the entire $𝜎$-algebra.

There are three ways of thinking on how the numerical values of probability manifest. Frequentism assigns values directly from data. Objectivism assigns values based on some model which is considered to describe aspects of the universe. Subjectivism characterizes probability as measuring beliefs about the world.

The definition of probability measure is as follows:

$\mathbb{P}(\mathrm{\Omega })=1$

and for any sequence of disjoint events in $\mathrm{ℱ︀}$:

$\mathbb{P}({\bigcup }_{𝑛=1}^{\infty }{𝐴}_{𝑛})={\sum }_{𝑛=1}^{\infty }\mathbb{P}({𝐴}_{𝑛})$

There are some other basic properties that arise from these definitions:

When $𝐴\subset 𝐵$

$\mathbb{P}(𝐵\setminus 𝐴)=\mathbb{P}(𝐵)-\mathbb{P}(𝐴)$

$\mathbb{P}(𝐴)\le \mathbb{P}(𝐵)$

$\mathbb{P}({𝐴}^{𝑐})=1-\mathbb{P}(𝐴)$

$\mathbb{P}(\varnothing )=0$

For all $𝐴,𝐵$:

$\mathbb{P}(𝐴\cup 𝐵)=\mathbb{P}(𝐴)+\mathbb{P}(𝐵)-𝑃(𝐴\cap 𝐵)$

Poincaré’s formula:

$\mathbb{P}({𝐴}_1\cup {𝐴}_2\cup \dots \cup {𝐴}_{𝑛})={\sum }_{𝐼\subset \left\{1,2,\dots ,𝑛\right\}}(-1{)}^{|𝐼|+1}\mathbb{P}({\bigcap }_{𝑖\in 𝐼}{𝐴}_{𝑖})$

$={\sum }_{𝑖=1}^{𝑛}\mathbb{P}({𝐴}_{𝑖})-{\sum }_{𝑖<𝑗\le 𝑛}\mathbb{P}({𝐴}_{𝑖}\cap {𝐴}_{𝑗})$

$+{\sum }_{𝑖<𝑗<𝑘\le 𝑛}\mathbb{P}\left({𝐴}_{𝑖}\cap {𝐴}_{𝑗}\cap {𝐴}_{𝑘}\right)-\dots +{\left(-1\right)}^{𝑛}\times \mathbb{P}\left({𝐴}_1\cap {𝐴}_2\dots \cap {𝐴}_{𝑛}\right)$

This is better expressed as the idea that the probability measure of the unions of sets can be expressed as the alternating signed partial sum of all the different partial intersections of the sequence of sets. It is proved by induction over the previous property.

Finite subadditivity (Boole’s inequality):

$\mathbb{P}\left({\bigcup }_{𝑖=1}^{𝑛}{𝐴}_{𝑖}\right)\le {\sum }_{𝑖=1}^{𝑛}\mathbb{P}({𝐴}_{𝑖})$

To show this, we shall set: ${𝐵}_1={𝐴}_1,{𝐵}_2={𝐴}_2\cap {𝐴}_1^{𝑐},\dots ,{𝐵}_{𝑛}={𝐴}_{𝑛}\cap {𝐴}_{𝑛-1}^{𝑐}\cap \dots \cap {𝐴}_1^{𝑐}$

You can think of this every successive element being disjoint with all the previous elements, which means

${𝐵}_{𝑖}\cap {𝐵}_{𝑗}=\varnothing$

and also

${\bigcup }_{𝑖=1}^{𝑛}{𝐵}_{𝑖}={\bigcup }_{𝑖=1}^{𝑛}{𝐴}_{𝑖}$

because the union of all the disjoint parts of the sets will include all the overlap in the sets.

A partition of a set $\mathrm{\Omega }$ is a collection of disjoint sets $\left\{{\mathrm{\Omega }}_{𝑖}\right\}$ where the union recreates the original set: ${\bigcup }_{𝑖}{\mathrm{\Omega }}_{𝑖}=\mathrm{\Omega }$

We can think of the probability of an event as the intersections of the event with a partition of $\mathrm{\Omega }$. This is the law of total probability. For $({𝐴}_{𝑖}{)}_{𝑖\in 𝐼}$ a partition of $\mathrm{\Omega }$ and every event $𝐵\in \mathrm{ℱ︀}$:

$\mathbb{P}(𝐵)={\sum }_{𝑖\in 𝐼}\mathbb{P}(𝐵\cap {𝐴}_{𝑖})$

Uniqueness of Probability Measures

Let’s consider the idea of a $𝜆$-system $\mathrm{𝒰︀}$ which is a family of events where:

$\mathrm{\Omega }\in \mathrm{𝒰︀}$

For any $𝐴,𝐵\in \mathrm{𝒰︀}$ and $𝐴\subset 𝐵$, $𝐵\setminus 𝐴\in \mathrm{𝒰︀}$

For a sequence of disjoint events ${𝐴}_{𝑛}$, indexed by $𝑛\in \mathrm{𝒰︀}$:

${\bigcup }_{𝑛}{𝐴}_{𝑛}\in \mathrm{𝒰︀}$

You might notice that this is fairly similar to the definition of a $𝜎$-algebra. In fact, a $𝜎$-algebra is a $𝜆$-system, and every $𝜆$-system closed to finite intersection is a $𝜎$-algebra.

Now let’s consider a collection of sets $\mathrm{ℳ︀}\subseteq 𝑃(\mathrm{\Omega })$ where $\mathrm{𝒰︀}(\mathrm{ℳ︀})$ is the smallest $𝜆$-system which contains $\mathrm{ℳ︀}$: $\mathrm{𝒰︀}(\mathrm{ℳ︀})=𝜎(\mathrm{ℳ︀})$

Note that this is distinct from the previous definition because although the $𝜆$-system is known to be a $𝜎$-algebra, it is now also defined to be the same as one which is generated by the same collection of sets.

Now let’s consider some probability measures ${\mathbb{P}}_1$ and ${\mathbb{P}}_2$ on the $𝜎$-algebra $\mathrm{ℱ︀}$ generated from a collection of sets $\mathrm{ℳ︀}$. If ${\mathbb{P}}_1$ and ${\mathbb{P}}_2$ coincide on $\mathrm{ℳ︀}$, then they coincide on $\mathrm{ℱ︀}$. This means that for every event, the two probability measures assign the same probabilities, which includes all the different kinds of events generated by the $𝜎$-algebra. This also applies to partitions of a sample space.

Monotone Class

This is a variant of a $𝜆$-system. A monotone class $\mathrm{ℳ︀}$ of subsets in $\mathrm{\Omega }$ is closed under the formation of monotone unions and intersections, i.e.

${𝐴}_1,{𝐴}_2,\dots \in \mathrm{ℳ︀}$ and ${𝐴}_{𝑛}\subseteq {𝐴}_{𝑛+1},{\cup }_{𝑛}{𝐴}_{𝑛}=𝐴\Rightarrow 𝐴\in \mathrm{ℳ︀}$

where the sets are increasing in size, containing the previous one,, and the union of all of them equals the last one.

${𝐴}_1,{𝐴}_2,\dots \in \mathrm{ℳ︀}$ and ${𝐴}_{𝑛}\supseteq {𝐴}_{𝑛+1}\Rightarrow {\cap }_{𝑛}{𝐴}_{𝑛}\in \mathrm{ℳ︀}$

or where the sets are decreasing in size, containing the next one, and the intersection of all of them equals the last one.

Open question: more stuff about monotone classes only if the professor talks about them.

Monotone Convergence Properties of Probability

How do we deal with infinity?

If ${𝐴}_{𝑛},𝐴\in \mathrm{ℱ︀}$ and ${𝐴}_{𝑛}\uparrow 𝐴$, which means that $𝐴$ is a monotonically increasing class, then $\mathbb{P}({𝐴}_{𝑛})\uparrow \mathbb{P}(𝐴)$ and the same is true for ${𝐴}_{𝑛}\downarrow 𝐴$.

Under countable subadditivity, the probability measure of the union of sets is less than or equal to the sum of the probability measures of those sets. That makes sense intuitively because it would be equal if the sets were disjoint, but the sets might overlap, making it less. We can use the above properties to extend this idea to countably infinite sets.

A null set is a set with probability measure 0. $\varnothing$ is the obvious example, and in infinite sets there may be others like the probability measure of a singleton set in the Borel $𝜎$-algebra.

Conditional Probability

“In some cases the probability of an event happening depends not just on the experiment itself but on other information as well. Conditional probability forms a framework in which this additional information can be incorporated.”

$\mathbb{P}(𝐴|𝐵)=\frac{\mathbb{P}(𝐴\cap 𝐵)}{\mathbb{P}(𝐵)}$

Obviously, the probability measure of $𝐵$ cannot be 0, which makes sense because in order for something to be conditioned, the condition has to have happened, which is not possible if the probability is 0.

We can get the multiplicative rule from this formula.

$\mathbb{P}(𝐴\cap 𝐵)=\mathbb{P}(𝐴|𝐵)\mathbb{P}(𝐵),$

$\mathbb{P}(𝐴\cap 𝐵\cap 𝐶)=\mathbb{P}(𝐴|𝐵\cap 𝐶)\mathbb{P}(𝐵|𝐶)\mathbb{P}(𝐶)$, etc.

Let’s take an example. We have Urn 1 which contains 2 black balls and 3 white balls, and Urn 2 which contains 1 of each. One urn is chosen at random and one ball is chosen randomly from that urn. If a black ball is chosen, what is the probability it came from Urn 1?

Let’s assign our variables.

$𝐴$ is the event of choosing Urn 1, and $𝐵$ is the event of choosing a black ball. We’re trying to find $𝑃(𝐴|𝐵)$, so we need $𝑃(𝐴\cap 𝐵)$. There are four total events in our sample space: { Urn1Black, Urn1White, Urn2Black, Urn2White }. The probability of Urn1Black is $0.5\times 0.4=0.2$. The probability of choosing a black ball is $0.5$ in the case of Urn 2 and $0.4$ in the case of Urn 1, which averages to $0.45$. So the overall probability is $0.2/0.45$, which is $4/9$.

The total probability formula given ${𝐴}_{𝑛}$ partition of $\mathrm{\Omega }$ is

$\mathbb{P}(𝐵)={\sum }_{𝑖=1}^{𝑛}\mathbb{P}(𝐵|{𝐴}_{𝑖})\mathbb{P}({𝐴}_{𝑖})$

for any event $𝐵\in \mathrm{ℱ︀}$.

If Stanley wants to draw a card from 20, where 8 are favorable to him, what is the probability of him drawing a favorable card if he draws first, second, or third?

First is easy. $8/20=0.4$

For second, there are two different possibilities. Either the first draw was a favorable or unfavorable card. Let’s use the formula above.

$𝐵$ is the event of Stanley drawing a favorable card. There are two $𝐴$ events in consideration, which are the first card being favorable and unfavorable. $\mathbb{P}({𝐴}_1)=2/5$ and $\mathbb{P}({𝐴}_2)=3/5$. If a favorable card was drawn first, then the new probability is $7/19$. Otherwise, it’s $8/19$. So the complete formula becomes

$\mathbb{P}(𝐵)=2/5\times 7/19+3/5\times 8/19=0.4$

It’s clear that the same situation would arise if he drew the third card.

Bayes’ formula is the following (like the others it’s based on a partition $𝐴$ and arbitrary set $𝐵\in \mathrm{ℱ︀}$):

$\mathbb{P}({𝐴}_{𝑗}|𝐵)=\frac{\mathbb{P}(𝐵|{𝐴}_{𝑗})\mathbb{P}({𝐴}_{𝑗})}{{\sum }_{𝑖=1}^{𝑛}\mathbb{P}(𝐵|{𝐴}_{𝑖})\mathbb{P}({𝐴}_{𝑖})}$

“This theorem can be interpreted as: All of a sudden we find out $𝐵$ happened. In the light of this new information there should be a simple way to update all the probabilities of the sets in the partition $(\mathbb{P}({𝐴}_{𝑗}|𝐵))$. Indeed, the Bayes rule provides just that.”

So you take the existing $𝐴$, and then condition it on $𝐵$, and then the calculation cascades across the events in the sample space i.e. $𝐴$.

Independence

Two events $𝐴$ and $𝐵$ are defined as independent if

$\mathbb{P}(𝐴\cap 𝐵)=\mathbb{P}(𝐴)\mathbb{P}(𝐵)$

which can be extended to the idea of being mutually independent (or simply independent) for a set of events:

$\mathbb{P}\left({\bigcup }_{𝑗\in 𝐽}{𝐴}_{𝑗}\right)={\prod }_{𝑗\in 𝐽}\mathbb{P}({𝐴}_{𝑗})$

There’s also the idea of jointly independent (or pairwise independent):

$\mathbb{P}({𝐴}_{𝑖}\cup {𝐴}_{𝑗})=\mathbb{P}({𝐴}_{𝑖})\mathbb{P}({𝐴}_{𝑗})$

which does not necessarily imply independence.

For two $𝜎$-algebras $𝐺$ and $𝐻$, they are $\mathbb{P}$-independent if

$\mathbb{P}(𝐺\cup 𝐻)=\mathbb{P}(𝐺)\mathbb{P}(𝐻)$

and the same applies for mutual independence as with events.

We can also show that if all $𝜎$-algebras in a family are each generated from a collection of independent sets, they are independent. This independence is also associative. There’s an additional property of “disassociativity of the independence”, where independent families included in bigger independent families remain independent.

$𝜎$-algebras generated by events are related in their independence.