Probability Spaces and Filtrations

Week 1

Readings

General Probability Theory

Infinite Probability Spaces

Let’s define an infinite probability space ${\mathrm{\Omega }}_{\infty }$. This sample space is uncountably infinite, which means that the probability of any particular outcome is zero. Therefore, in order to determine the probability of an event, we define the formula for the probability of certain simple events, then use probability measures to determine the probability of more complicated events.

$𝜎$-algebras form the basis of these simple events:

Lebesgue Measures

The probability of picking a number in between ⅓ and ⅔ is ⅔ - ⅓ = ⅓. This is also true for open intervals because the probability of an individual point occurring is 0.

We can apply $𝜎$-algebras to the Lebesgue measure by generating a $𝜎$-algebra from the collection of all closed intervals in our interval, like:

$[0,\frac{1}{3}],[\frac{1}{2},1],[\frac{1}{4},\frac{230}{243}]$

and so on. Open intervals can be defined as a union of a sequence of closed intervals, so these are included as well. The $𝜎$-algebra generated by this process is the Borel $𝜎$-algebra, specifically of the subset $[0,1]$, which is $\mathrm{ℬ︀}[0,1]$. The Borel $𝜎$-algebra is a well-defined set of events (remember, an event is a set) where the outcome is a real number. In this context, under the rules of $𝜎$-algebras, an event is an interval, a union of intervals, a complement of an interval, or any combination of these.

Coin Tosses

Let’s think about coin tosses. Through filtration (see notes), we can define the probability of every event that can be described as a finite number of coin tosses. We can determine the probability of every individual sequence in ${\mathrm{\Omega }}_{\infty }$ by showing that their probabilities converge to zero since they are fractions that multiply together infinitely.

We can use the rules of $𝜎$-algebras and probability measures to determine the probability of each of these events, like how the probability measure of the union of disjoint events (remember, events are sets) is equivalent to the summation of the probability measures of each event.

There are some events that are not easily computed unless we define them in this respect. Consider

${lim}_{𝑛\to \infty }\frac{{𝐻}_{𝑛}({𝜔}_1\dots {𝜔}_{𝑛})}{𝑛}=\frac{1}{2}$

which describes all sequences of tosses where the long-run probability of choosing heads is ½. This set must be in ${\mathrm{ℱ︀}}_{\infty }$ so we can determine its probability measure. If we choose positive integers $𝑚$ and $𝑛$, we can define

${𝐴}_{𝑛,𝑚}=\left\{𝜔;|\frac{{𝐻}_{𝑛}({𝜔}_1\dots {𝜔}_{𝑛})}{𝑛}-\frac{1}{2}|\le \frac{1}{𝑚}\right\}$

where this set is in ${\mathrm{ℱ︀}}_{𝑛}$ i.e. that step in the filtration. We can define this as a combination of sets if we recognize that for each threshold $𝑚$, there is an $𝑁$ where for all $𝑛$ after $𝑁$ i.e. for all sufficiently long sequences after a certain point, the average of the sequence meets the threshold, following the definition of a limit. By considering intersections as “all of these are true” and unions as “at least one must be true”, we can define this as

$𝐴={\bigcap }_{𝑚=1}^{\infty }{\bigcup }_{𝑁=1}^{\infty }{\bigcap }_{𝑛=𝑁}^{\infty }{𝐴}_{𝑛,𝑚}$

By using the law of large numbers (large enough sequences will converge to the true value), we can see that $\mathbb{P}(𝐴)=1$ for probability of ½, i.e. that the probability of an outcome belonging to this set is 1

Note here that this is not the same thing as saying that every outcome occurs in this set. We can easily imagine outcomes that do not. However, any individual event has a probability of 0 of occurring, so all those exceptional events together have a probability of 0. The way we describe this technically is that the event corresponding to the set with probability measure of 1 occurs almost surely.

Random Variables and Distributions

Let’s define a random variable in the context of all this measure theory. We’ll generalize the random variable to a function $𝑋$ with the property that (in probability space $(\mathrm{\Omega },\mathrm{ℱ︀},\mathbb{P})$

$\left\{𝑋\in 𝐵\right\}=\left\{𝜔\in \mathrm{\Omega };𝑋(𝜔)\in 𝐵\right\}$

where $𝐵$ is a Borel subset of $ℝ$, this subset of $\mathrm{\Omega }$ is in $𝜎$-algebra $\mathrm{ℱ︀}$. Essentially, this defines $𝑋$ as a function mapping outcome $𝜔$ to a Borel subset i.e. an interval or combination of intervals on the real number line. This is the formal way to turn probability measures into actual numbers (see dice example from lecture notes).

We are not interested in $𝑋$ taking a specific value as that will typically be 0, so we will usually talk about the probability that $𝑋\in 𝐵$.

Stock Prices

Let’s consider the binomial asset pricing model where ${𝑆}_0(𝜔)=4$, $𝑢=2$, and $𝑑=1/2$ , which is generalized as

i.e. the stock price doubles on heads and halves on tails. Let’s think of the probability of the stock price now as the distribution of mass. ${𝑆}_0=4$ occurs with probability one, so this random variable has a unit of mass on 4. If we move two step forwards, $\mathbb{P}\left\{{𝑆}_2=16\right\}={𝑝}^2$, $\mathbb{P}\left\{{𝑆}_2=4\right\}=2𝑝𝑞$ and $\mathbb{P}\left\{{𝑆}_2=1\right\}={𝑞}^2$, where $𝑝$ and $𝑞$ are the probabilities of $𝑢$ and $𝑑$, respectively. There are three lumps of mass distributed between 16, 4, and 1, each with the size of their probabilities. Now let’s think of spreading that mass continuously. The distribution of a random variable is about the amount of mass in a set. This distribution measure ${𝜇}_{𝑋}$ of random variable $𝑋$ is a probability measure $𝑃$, but it is done on $𝐵$ rather than subsets of $\mathrm{\Omega }$. In other words, it measures the probability of $𝑋$ being in a particular Borel subset.

Random variables have distributions, but they are not the same thing. The variable is the mapping between the outcome and the real number line i.e. a function, and the distribution is a measure of the probability that the variable is a certain value, or more generally, in a certain Borel subset. A random variable can have two different distributions and two different random variables can have the same distribution. Consider $\mathbb{P}$ as the uniform measure of $[0,1]$. We have two random variables $𝑋(𝜔)=𝜔$ and $𝑌(𝜔)=1-𝜔$. The distribution measure of $𝑋$ is uniform, which means that all outcomes are equally likely. The distribution measure of $𝑌$ is also uniform, despite being different from $𝑋$, they’re both $𝑏-𝑎$.

Now instead let’s define $\mathbb{P}$ as

$\mathbb{P}[𝑎,𝑏]={\int }_{𝑎}^{𝑏}2𝜔𝑑𝜔={𝑏}^2-{𝑎}^2,0\le 𝑎\le 𝑏\le 1$

This is still a probability measure on $[0,1]$ because $\mathbb{P}[0,1]=1$. If you take a Borel subset of this interval, you can calculate its probability measure through the properties of probability measures applied to each interval that is unioned or complemented.

The distribution measure is one way to record a distribution. The other is the cumulative distribution function $𝑐𝑑𝑓$

$𝐹(𝑥)=\mathbb{P}\left\{𝑋\le 𝑥\right\},𝑥\in ℝ$

The difference is that distribution measure is a direct mapping of the idea of probability measure onto the Borel subset, while the cdf is a distribution measure on the specific Borel subset ${𝜇}_{𝑋}(-\infty ,𝑥]$. It can also be converted backwards as ${𝜇}_{𝑋}(𝑥,𝑦]=𝐹(𝑦)-𝐹(𝑥)$

Through the rules of probability measures, when the distribution measure is determined for every interval $[𝑎,𝑏]$ in $ℝ$, then it is determined for every Borel subset of $ℝ$.

We can also get more detail on the distribution in two special functions.

We can have a density function $𝑓(𝑥)$

${𝜇}_{𝑋}[𝑎,𝑏]=\mathbb{P}\left\{𝑎\le 𝑋\le 𝑏\right\}={\int }_{𝑎}^{𝑏}𝑓(𝑥)𝑑𝑥$

which describes the distribution through its integral.

The probability mass function tells us that a random variable $𝑋$ takes one of the values in a particular sequence with probability 1. You could imagine, for instance, if $𝑋={𝜔}^2$, there is a probability of 1 that $𝑋$ is one of the values in the sequence of squared numbers. Strictly, this is defined where ${𝑝}_{𝑖}=\mathbb{P}\left\{𝑋={𝑥}_{𝑖}\right\}$ and ${\sum }_{𝑖}{𝑝}_{𝑖}=1$.

The sequence of numbers ${𝑥}_1,{𝑥}_2,\dots$ is a Borel subset.

Random variables can be described by a density, a probability mass function, both, or neither. This last one is very advanced.

Let’s define the standard normal density:

$𝜑(𝑥)=\frac{1}{\sqrt{2𝜋}}{𝑒}^{-\frac{{𝑥}^2}{2}}$

and the cumulative normal distribution function as

$𝑁(𝑥)={\int }_{-\infty }^{𝑥}𝜑(𝜉)𝑑𝜉$

This function is strictly increasing and maps $ℝ$ to $(0,1)$ and has a strictly increasing inverse function ${𝑁}^{-1}(𝑦)$. Let’s consider the uniformly distributed random variable $𝑌$ on probability space $(\mathrm{\Omega },\mathrm{ℱ︀},\mathbb{P})$, and $𝑋={𝑁}^{-1}(𝑌)$.

The distribution measure is the probability measure that $𝑋(𝜔)$ is in the interval defined. The distribution measure of $𝑌𝜔$ therefore is $(𝑁(𝑎),𝑁(𝑏)$, which means that the distribution measure is $𝑁(𝑏)-𝑁(𝑎)$ because $𝑌$ is uniformly distributed, which is the same as

${\int }_{𝑎}^{𝑏}𝜑(𝑥)𝑑𝑥$.

This distribution measure ${𝜇}_{𝑋}$ is known as the standard normal distribution, and any random variable with this distribution is a standard normal random variable. This technique of generating a standard normal random variable is called the probability integral transform and it is essential to Monte Carlo simulation.

A simpler way to get a standard normal random variable is to take this integral from arbitrary $𝑎$ and $𝑏$ as a probability measure for the random variable $𝑋(𝜔)=𝜔$. This method works for any distribution, but it is not useful when we have multiple random variables with their own distributions and dependencies. For those cases we need a single probability space.

Expectation

We are interested in computing an “average value” of $𝑋$ in a probability space. For a finite $\mathrm{\Omega }$, this is trivially:

$𝔼𝑋={\sum }_{𝜔\in \mathrm{\Omega }}𝑋(𝜔)\mathbb{P}(𝜔)$

The same can be done for a countably infinite $\mathrm{\Omega }$.

However, if $\mathrm{\Omega }$ is uncountably infinite, expectation must be defined as an integral.

First, let’s consider the Riemann integral. This involves partitioning the area under the function in ever smaller subintervals and adding them up, which gets closer and closer to the true integral. There is an upper Riemann sum based on the maximum of the areas and lower based on the minimum, which both converge to the Riemann integral.

The reason we can’t use the Riemann integral for expectation is that our x-axis is $\mathrm{\Omega }$, which is not necessarily a subset of $ℝ$. We instead must partition the y-axis.

We have defined this as a Lebesgue sum, which is the sum of

Lecture Notes

$𝜎$-algebras

The set of possible outcomes is the sample space $\mathrm{\Omega }$. If you flip a coin, the sample space is $\left\{𝐻,𝑇\right\}$

We are interested in the probability of events, which might be more complicated than an outcome. A $𝜎$-algebra is a set of elements that belong to $\mathrm{ℱ︀}$ the powerset of $\mathrm{\Omega }$. The empty set must belong to it, all complements of sets must belong there, and the unions of all the composite sets must also belong there.

The $𝜎$-algebra MUST contain $\varnothing$ and $\mathrm{\Omega }$, and this particular algebra is called the trivial $𝜎$-algebra because it is the simplest possible one. This might be confusing because $𝜎$ can represent both the standard deviation and the algebra, so use your brain for context.

A fancier $𝜎$-algebra could be generated from a set $𝐴$ contained in it, which would be $𝜎(𝐵)=\left\{\varnothing ,𝐴,\mathrm{\Omega },{𝐴}^{𝑐}\right\}$

Trick question: if I have a set $𝐵$, what is it? It’s actually the same just with $𝐵$

A different question is $𝜎(𝐴,𝐵)$. This would start with generators $\left\{\varnothing ,\mathrm{\Omega },𝐴,𝐵,{𝐴}^{𝑐},{𝐵}^{𝑐},𝐴\cup 𝐵,(𝐴\cup 𝐵{)}^{𝑐}\right\}$

The number of elements in $𝜎$-algebra can be found as 2 to the power of the number of distinct regions. For disjoint $𝐴$ and $𝐵$, this is 3.

What about joint $𝐴$ and $𝐵$ like a Venn diagram. There’s the area outside of $𝐴$ and $𝐵$, the area just in $𝐴$, the area just in $𝐵$, and the area shared by $𝐴$ and $𝐵$, so 4, which means 16 elements.

$𝜎(𝐴,𝐵)=\{\varnothing ,\mathrm{\Omega },𝐴,𝐵,{𝐴}^{𝑐},{𝐵}^{𝑐},𝐴\cup 𝐵,(𝐴\cup 𝐵{)}^{𝑐},$

$𝐴\cup {𝐵}^{𝑐},(𝐴\cup {𝐵}^{𝑐}{)}^{𝑐}={𝐴}^{𝑐}\cap 𝐵=𝐵\backslash 𝐴,𝐵\cup {𝐴}^{𝑐},(𝐵\cup {𝐴}^{𝑐}{)}^{𝑐}=𝐴\backslash 𝐵,{𝐴}^{𝑐}\cup {𝐵}^{𝑐},({𝐴}^{𝑐}\cup {𝐵}^{𝑐}{)}^{𝑐}=𝐴\cap 𝐵,$

$\left({\left(𝐵\cup {𝐴}^{𝑐}\right)}^{𝑐}\right)\cup \left({\left(𝐴\cup {𝐵}^{𝑐}\right)}^{𝑐}\right)\}$

Open question: draw out a venn diagram of what each of these mean

Let’s say we choose some random person at Stevens. Event A is undergraduate students, Event B is graduate students. Some people are neither (professors) and some people are both (accelerated students). The $𝜎$-algebra expresses the event space.

Probability Measure

The function mapping $\mathrm{ℱ︀}$ maps all of the sets in our $𝜎$-algebra to events in the event space, which we can use for the probability measure $\mathbb{P}$. We have two axioms:

1. $\mathbb{P}(\mathrm{\Omega })=1$
2. the $\mathbb{P}$ of a sequence of disjoint sets is equal to the sum of the

The probability space is the combination of the sample space, function mapping, and probability measure: $(\mathrm{\Omega },\mathrm{ℱ︀},\mathbb{P})$

If $\mathbb{P}(𝐴)=1$, then the event $𝐴$ occurs almost surely. Doesn’t mean it can’t happen, but it has zero probability of occurring.

Example: If I were to pick a completely random real number between 0 and 1, the probability I would pick a rational number is 0, even though there clearly are rational numbers that exist.

A random variable is a function $𝑋$ that maps from the event space to all real numbers $ℝ$. You’ve been doing this your whole life without knowing it. Roll a six-sided die. What’s the sample space? It’s not 1, 2, 3, 4, 5, 6, it’s a set of weird looking symbols that we map onto numbers. You could equally create a $𝑌$ which says 6, 7, 8, 9, 10, 11.

The Borel subsets are every possible subdivision of the number line. It’s essentially a way to describe the real number line through set theory.

Expectation

Regular expectation is the probability of each outcome added together. How can you be certain that you’re exhaustive? We have to define elements in our sample space, and we use set theory to derive this:

$𝔼[𝑋]={\sum }_{𝜔\in \mathrm{\Omega }}𝑋(𝜔)\mathbb{P}(𝜔)$

This looks like regular expectation, but it can be extended to understand the expectation of a function on a random variable as well.

Assume we have a fair coin. $𝑋(𝜔)$ is 1 if $𝜔$ is heads, 0 if tails. $𝑌(𝜔)$ is 10 and −2 for the same.

What is $𝔼(𝑋)$? ${\sum }_{𝜔\in \mathrm{\Omega }}𝑋(𝜔)\mathbb{P}(𝜔)=𝑋(ℎ)\mathbb{P}(ℎ)+𝑋(𝑡)\mathbb{P}(𝑡)=1\ast 0.5+0\ast 0.5=0.5$

Let’s think more advanced:

$𝔼[{𝑒}^{𝑥}]=𝔼[𝑔(𝑥)]={\sum }_{𝜔\in \mathrm{\Omega }}𝑔(𝑋)$

Expected value are linear operators, so $𝔼[𝛼𝑋+𝛽𝑌]=𝛼𝔼[𝑋]+𝛽𝔼[𝑌]$ this is very useful for operations

Jensen’s inequality mainly communicates that Variance is never negative: if $𝜑$ is a convex real-valued function and if $𝔼[𝑋]<\infty$, then

$𝜑(𝔼[𝑋])<=𝔼[𝜑(𝑋)]$

Probability being greater than 1 and variance being negative are massive red flags that should stop you in your tracks.

Change of Measure

We can shift between probability spaces and get different probability measures like so:

$𝑍(𝜔)=\frac{\tilde{\mathbb{P}}(𝜔)}{\mathbb{P}(𝜔)}$

this is random variable which is almost surely nonnegative with $𝔼[𝑍]=1$.

new function:

$\tilde{\mathbb{P}}(𝐴)={\int }_{𝐴}𝑍(𝜔)𝑑\mathbb{P}(𝜔)$

so $\tilde{\mathbb{P}}$ is a probability measure.

We could consider the same thing for $\mathrm{\Omega }$ as the set instead of $𝐴$, but the integral of $𝑍$ with respect to the probability measure over the sample space is just the same thing as $𝔼[𝑍]$, which is axiomatically 1.

$𝑣𝔼[𝑋𝑍]={\int }_{\mathrm{\Omega }}𝑋(𝜔)$

We know from the definition of $𝑍$ as the ratio of probability measure that $𝑍(𝜔)𝑑\mathbb{P}(𝜔)=𝑑\tilde{\mathbb{P}}$ which gives us our integral over $\mathrm{\Omega }$ and $𝔼[𝑍]$

Equivalent Measure

Equivalent probability measures only need to agree on which sets in $\mathrm{ℱ︀}$ have probability zero. Let’s say we’ve bent a coin so much that now landing on heads has a 10% chance. The event space is still heads and tails, but the probability measure is different because the probability of the heads event and the tail event are different. Yet, these are still equivalent because they agree that the only set with probability zero is $\varnothing$.

Let’s say you’re drawing a card from a deck and checking the suit. If you remove some cards, there’s still an equivalent probability measure. But if you remove all the clubs, it is not equivalent.

Radon-Nikodým derivative Open question: add this from slides

Filtration

A filtration is a collection of $𝜎$-algebras where each one contains all the previous ones, like Russian nesting dolls. The best way to think about them is information that accrues over time at each step.

If we have random variable $𝑋$, the $𝜎$-algebra generated by $𝑋$ is $𝜎(𝑋)$, which is the collection of all subsets of $\mathrm{\Omega }$ where $(𝑋\in 𝐵)$ for $𝐵$ Borel subset Open question: look up this up more formally.

Flipping a coin:

$\mathrm{\Omega }=\left\{{𝜔}_1,{𝜔}_2,{𝜔}_3:{𝜔}_{𝑖}\in \left\{ℎ,𝑡\right\}\right\}$

Before we even flip a coin, what do we know we would know about the coin flip? It’s either heads or tails.

$\mathrm{ℱ︀}(1)=\left\{\varnothing ,\mathrm{\Omega },𝐻,𝑇\right\}$

What if we flip it twice? We have four distinct outcomes now, HH, HT, TH, and TT.

$\mathrm{ℱ︀}(2)=\left\{\varnothing ,\mathrm{\Omega },𝐻𝐻,𝐻𝑇,𝑇𝐻,𝑇𝑇,𝐻{𝐻}^{𝑐},𝐻{𝑇}^{𝑐},𝑇{𝐻}^{𝑐},𝑇{𝑇}^{𝑐},𝐻𝐻\cup 𝐻𝑇,𝐻𝐻\cup 𝑇𝐻,𝐻𝐻\cup 𝑇𝑇,𝐻𝑇\cup 𝑇𝐻,𝐻𝑇\cup 𝑇𝑇,𝑇𝐻\cup 𝑇𝑇\right\}$

What about three times? 😅

$\mathrm{ℱ︀}(3)=\left\{\varnothing ,\mathrm{\Omega },\dots \right\}$ (256 elements)

Measurability

If $\mathrm{𝒢︀}$ is also a $𝜎$-algebra, and every set in $𝜎(𝑋)$ is in $\mathrm{𝒢︀}$, then $𝑋$ is said to be $\mathrm{𝒢︀}$-measurable. If we have

$\mathrm{\Omega }=\left\{𝑎,𝑏,𝑐,𝑑\right\}$,

and $𝑋(𝜔)$ 2 if a, 0, if b or c, −2 if d

and $𝑌(𝜔)$ 13 if a or b, −2 if c or d

and $𝑍(𝜔)$ −2 if a or d, 3 if b or c

What is $𝜎(𝑍)$? In order to generate a $𝜎$-algebra from a random variable, we look at the partitions in the function and add those into the generator:

$𝜎(𝑍)=\left\{\varnothing ,\mathrm{\Omega },\left\{𝑎,𝑑\right\},\left\{𝑏,𝑐\right\}\right\}$

We can also more easily define these sets as $𝛼$, $𝛽$, $𝛾$, etc. to simplify writing down.

We can see that $𝑋$ is $𝑍$-measurable. But what does that actually mean?

Let’s say I know $𝑋=2$. I do know the values of $𝑌$ and $𝑍$, because the outcome has to be $𝑎$.

But if $𝑋$ is 0, I have no idea what $𝑌$ is. I do know what $𝑍$ is.