Continuous Random Variables and Transformations

Lecture Notes

Recall for a discrete random variable. The moment formula is

𝔼[𝑋]=∑𝑥∈𝑋(Ω)𝑥ℙ(𝑋=𝑥)

The moment for a continuous rv is the integral

𝔼[𝑋]=∫−∞∞𝑥𝑓(𝑥)𝑑𝑥

All the possibilities are the real line here.

Most of today’s class will deal with integrals. We are going to calculate a lot of these, just like a single variable calculus class. We will also learn multivariable calculus for multiple variables.

Generalization of expectation for any function is definition 5.9 (skip technical assumptions)

𝔼[ℎ(𝑋)]=∫−∞∞ℎ(𝑥)𝑓(𝑥)𝑑𝑥

You would just compute the antiderivative of ℎ(𝑥) and then evaluate the integral by subtracting the top from bottom. For an indicator variable, it is 0 to 1 not -infinity to infinity.

Now I want to go further, I want to know the full distribution, not just the expectation, of ℎ(𝑋)

This is given by theorem 5.10 which tells us how to determine the probability density function. The PDF of 𝑌=ℎ(𝑋) is given by

𝑓𝑌(𝑦)=1ℎ′(ℎ−1(𝑦))𝑓(ℎ−1)(𝑦)𝐼{𝑦∈ℎ(𝐷)}

Here, 𝑓(ℎ−1) denotes the composition of f on inverse h. First I apply the inverse h, then I apply the f. Sometimes you use the notation 𝑓∘ℎ−1

How do you prove this? We will sketch. We will make the assumption that ℎ is non-decreasing.

ℙ(𝑌≤𝑦)=ℙ(ℎ(𝑋)≤𝑦)=ℙ(𝑋≤ℎ−1(𝑦))

We can do the inverse because it’s non-decreasing.

This is true, so we can write this probability as the integral of the PDF of X

=∫−∞ℎ−1(𝑦)𝑓(𝑥)𝑑𝑥

If I want to say the PDF of the random variable 𝑌, then it gets the 𝑌 subscript; without the subscript, it’s for 𝑋.

Now we will do a change of variables

𝑥=ℎ−1(𝑧)

We know that 𝑥 is inside the integral, so we will write the equivalence that when it is in between those two values, it is equivalent to the event that 𝑧 is in between −∞ and 𝑦.

This is important because we will change the integration range:

=∫−∞𝑦𝑓(ℎ−1(𝑧))𝑑ℎ−1(𝑧)

You can move this part to the front

=∫−∞𝑦𝑓(ℎ−1(𝑧))⋅|(ℎ−1)′(𝑧)|𝑑𝑧

We will see that

𝑓𝑌(𝑦)=𝑑𝑑𝑦ℙ(𝑌≤𝑦)

Now if we take the derivative of this integral, then we get

𝑑𝑑𝑦∫−∞𝑦𝑓(ℎ−1(𝑧))|(ℎ−1)′(𝑧)|𝑑𝑧

We get something like the fundamental theorem of calculus here, or part of it, or Leibniz rule. you have the following result: 𝑑𝑑𝑥(∫−∞𝑥𝑓(𝑢)𝑑𝑢)=𝑓(𝑥)

This identity is learned in calculus.

So, we can show that everything inside doesn’t depend on 𝑦, so we can just evaluate the integral with it.

=𝑓(ℎ−1(𝑦))|(ℎ−1)′(𝑦)|

Then we will use this result, from a second theorem of calculus called the implicit function theorem. This is more real analysis, here’s a simple example.

ℎ−1(ℎ(𝑦))=𝑦

Now the idea is we want to do differentiation on both sides.

𝑑𝑑𝑦(ℎ−1(ℎ(𝑦))=𝑑𝑑𝑦(𝑦)

Because we have this equality, the derivative preserves the equality (maybe not integration because of range). Applying the chain rule, we get

(ℎ−1)′(ℎ(𝑦))=1ℎ′(𝑦)

And then we do a change of variable.

𝑦=ℎ−1(𝑢)

y is just a dummy variable, so we can apply this here, to finally show that

=𝑓(ℎ−1(𝑦))|1ℎ′(ℎ−1(𝑦))| Interesting corollary 5.11 that if X has pdf f and we have a Y which is aX + b, then pdf of Y is

𝑓𝑌(𝑦)=1|𝑎|𝑓(𝑦−𝑏𝑎)

This is a special case when the ℎ(𝑥) is 𝑎𝑋+𝑏, can be shown by differentiation. All you need to find is ℎ′(𝑥) and ℎ−1(𝑦)

We can essentially think of 𝑌 as a linear combination of a different random variable whose pdf is known, and then we can derive the pdf from the existing pdf f.

5.4.1

We are concerned with a uniform random variable defined continuously on [𝑎,𝑏] on ℝ

Definition 5.13 says that the pdf of this is

𝑓(𝑥)=1𝑏−𝑎,𝑥∈[𝑎,𝑏]

We will use the notation to indicate uniform distribution

𝑥∼𝑈[𝑎,𝑏]

Let’s do something non-trivial. Proposition 5.15 says that if we have 𝑋∼𝑈[𝑎,𝑏], then

𝔼[𝑋]=𝑎+𝑏2

𝕍[𝑋]=(𝑏−𝑎)212

How do we prove? We can create the integral based on the definition

We have from algebra that

𝑏3−𝑎3=(𝑏−𝑎)(𝑎2+𝑎𝑏+𝑏2)

which we can use to prove the variance.

We are generally solving for

1𝑏−𝑎∫𝑎𝑏𝑓(𝑥)𝑑𝑥

and that’s how we solve for expectation, for 𝑓(𝑥) being 𝑥 or 𝑥2

We have to turn these types of problems into a pdf, that will be done on the midterm.

What is the cdf for a uniform variable?

By proposition 5.16, it’s

𝐹𝑋(𝑡)={0if 𝑡<𝑎𝑡−𝑎𝑏−𝑎if 𝑡∈[𝑎,𝑏]1if 𝑡>𝑏

This is the same as ℙ(𝑋≤𝑡)

We can accomplish this case conclusion by the integral

=∫𝑎𝑡1𝑏−𝑎𝐼[𝑎,𝑏](𝑢)𝑑𝑢 Very simple proof for this.

Now let’s consider a simple example 5.8 where 𝑋∼𝑈[0,1], we will compute ℙ(14≤𝑥≤34)

To solve, this we will solve as an integral

=∫1434𝑓(𝑥)𝑑𝑥=∫14341𝑑𝑥=12

This is one way to solve this, or you could separate it into cdf up to ¾ and ¼.

5.4.2

The next thing is the exponential distribution, definition 5.17. We introduce the auxiliary parameter 𝜆>0, then

𝑓(𝑥)=𝜆𝑒−𝜆𝑥,𝑥>0,𝑋∼Exp(𝜆)

That is what is meant when we are given a distribution. How do we verify that this result is a pdf? Proposition 5.18 says

∫−∞∞𝑓(𝑥)𝑑𝑥

The range of integration starts from 0

=𝜆∫0∞𝑒−𝜆𝑥𝑑𝑥

Now we can do some calculus

𝑑𝑑𝑥𝑒−𝜆𝑥=𝑒−𝜆𝑥(−𝜆)

𝑒−𝜆𝑥=𝑑𝑑𝑥𝑒−𝜆𝑥−𝜆

𝑒−𝜆𝑥𝑑𝑥=𝑑(𝑒−𝜆𝑥−𝜆)

Now we can do integration by parts, which is a very important technique for this chapter that we will do over and over again.

If we are interested in integrating

𝜆𝑓(𝑥)𝑑𝑔(𝑥)=𝑓(𝑥)⋅𝑔(𝑥)−𝜆𝑔(𝑥)−∫𝑔(𝑥)𝑑𝑓(𝑥)

And this is the process of integration by parts.

(We don’t actually need integration by parts here, but we will need to do it for the moments)

Now we will show the moments by proposition 5.19

𝔼[𝑋]=1𝜆

𝕍[𝑋]=1𝜆2

We will now try to evaluate this integral

=𝜆∫0∞𝑥𝑒−𝜆𝑥𝑑𝑥

And this is where we need integration by parts

=𝜆∫0∞𝑥𝑑𝑒−𝜆𝑥−𝜆 First we take the product

=𝜆(𝑥𝑒−𝜆𝑥−𝜆|𝑥=0𝑥=∞−∫0∞𝑒−𝜆𝑥−𝜆𝑑𝑥)

Variance is not easy because you need to integrate twice by parts.

𝔼[𝑋2]=∫0∞𝑥2𝜆𝑒−𝜆𝑥𝑑𝑥=𝜆∫0∞𝑥2𝑒−𝜆𝑥𝑑𝑥=𝜆∫0∞𝑥2𝑑(𝑒−𝜆𝑥−𝜆)

We will try now to get rid of that denominator to get

=−∫0∞𝑥2𝑑𝑒−𝜆𝑥

Integration by parts:

−𝑥2𝑒−𝜆𝑥|𝑥=0𝑥=∞−∫0∞𝑒−𝜆𝑥𝑑𝑥2

=−(0−∫0∞𝑒−𝜆𝑥2𝑥𝑑𝑥)

=2∫0∞𝑒−𝜆𝑥𝑥𝑑𝑥

And we already have shown this result so it is

=2𝜆2

And then after subtracting by the expectation squared you get

𝕍[𝑋]=1𝜆2

What is the cdf of the exponential random variable? Proposition 5.20 will tell us that

𝐹𝑋(𝑥)=1−𝑒−𝜆𝑥

Here is the proof, which is just calculation

𝐹𝑋(𝑥)=∫0𝑥𝑓(𝑢)𝑑𝑢=∫0𝑥𝜆𝑒−𝜆𝑢𝑑𝑢

The difference between a CDF and a statistical measure is that we’re measuring up to (x), not infinity, but the technique is the same.

=𝜆∫0𝑥𝑑(𝑒−𝜆𝑢−𝜆)=𝜆𝑒−𝜆𝑢−𝜆|𝑢=0𝑢=𝑥

There is a memoryless property of the exponential distribution.

ℙ(𝑋>𝑡+𝑠|𝑋>𝑡)=ℙ(𝑋>𝑠),𝑠,𝑡≥0

I know that my exponential variable is bigger than t. There is some interval between t and t + s which is length s. I know my random variable 𝑋 is bigger than t + s, given that it is bigger than t. But actually it doesn’t really matter what the probability is between t and t + s, there is no memory of the starting point. You only care about the probability after s. (We will not prove this)

Let’s take example 5.9, it will be easy to prove the cdf

𝑓𝑋(𝑡)=1100𝑒−𝑡100,𝜆=1100

Then we get expectation by the formula 1𝜆 How do we solve if we are given probability between an interval like

ℙ(0≤𝑋≤50)

We take the integral between these two points.

=∫0501100𝑒−𝑡100𝑑𝑡

And we can actually do a change of variable here from

𝑠=𝑡100

Which will give us

=∫012𝑒−𝑠𝑑𝑠=𝑒−𝑠−1|𝑠=0𝑠=12=𝑒−12−1−1=1−𝑒−12

5.4.3

The normal distribution by definition 5.22 has the distribution function

𝑓(𝑥)=12𝜋𝜎2𝑒−(𝑥−𝜇)22𝜎2,𝑥∈ℝ

denoted by

𝑋∼𝑁(𝜇,𝜎2) We have to show that the normal distribution is a PDF, which is not very straightforward.

Proposition 5.24

Proposition 5.25 tells us that the expected value and variance are

𝔼[𝑋]=𝜇

𝕍[𝑋]=𝜎2

which is very convenient for us. But we will need to formally verify this

We will copy this pdf into the integral

𝔼[𝑋]=12𝜋𝜎2∫−∞∞𝑥𝑒−(𝑥−𝜇)22𝜎2𝑑𝑥

And we are going to do a change of variable

𝑦=𝑥−𝜇

We have the same integration region because they are infinite. We can replace x in this whole function.

=12𝜋𝜎2∫−∞∞(𝑦+𝜇)𝑒−𝑦22𝜎2𝑑𝑦

We will separate the integral into two parts

=12𝜋𝜎2∫−∞∞𝑦𝑒−𝑦22𝜎2𝑑𝑦+𝜇∫−∞∞𝑒−𝑦22𝜎2𝑑𝑦

The first integrand is an odd function. This means that 𝑓(𝑥)=−𝑓(−𝑥). Even functions are 𝑓(𝑥)=𝑓(−𝑥). For any integral from -infinity to infinity, we can combine all the negatives and all the positives. And if it is symmetric around 0 with the opposite, which is the meaning of odd, then when they’re added it should give you 0.

Then we can recognize that for the second integrand that it must be equal to 𝜇 by the fact that the normal distribution area is equal to 1 (assumed).

Now for the variance. By definition, because we know the expectation already, we get

𝕍[𝑋]=𝔼[(𝑋−𝜇)2]

which by the pdf is

=12𝜋𝜎2∫−∞∞(𝑥−𝜇)2𝑒−(𝑥−𝜇)22𝜎2𝑑𝑥

Then we can do a change of variable, and then solve via polar coordinates on page 138

=12𝜋𝜎2∫−∞∞𝑦2𝑒−𝑦22𝜎2𝑑𝑦

Remark 5.2.7 says that if 𝑋∼𝑁(0,1), then 𝜇+𝜎𝑋∼𝑁(𝜇,𝜎2)

The inverse operation

𝑌∼𝑁(𝜇,𝜎2),𝑌−𝜇𝜎∼𝑁(0,1)

is known as standardization.

Proof: if you let

𝑌=𝜇+𝜎𝑋=ℎ(𝑋)

Using the corollary 5.11 for linear combination we can prove this.

Useful property of normal distribution. If we have two normal distributions 𝑋 and 𝑌 that are independent, then

𝑋+𝑌∼𝑁(𝜇1+𝜇2,𝜎12+𝜎22) We have to do a convolution to solve this, involves integration between the two things.

5.4.14

Last one for today is the gamma distribution.

We need some preparation, in the gamma function.

Γ(𝑥)=∫0∞𝑡𝑥−1𝑒−𝑡𝑑𝑡

Proposition 5.30 says the properties of the gamma function. We have the property (a) that

Γ(𝑥+1)=𝑥Γ(𝑥),𝑥>0

and (b) that

Γ(𝑛+1)=𝑛!

and (c), which the source note does not prove,

Γ(12)=𝜋

Proof of (a) is that

=∫0∞𝑡𝑋𝑒−𝑡𝑑𝑡=∫0∞𝑡𝑋(𝑑𝑒−𝑡−1)

𝑡𝑥𝑒−𝑡−1|𝑡=0𝑡=∞−∫0∞𝑒−𝑡−1𝑑(𝑡𝑥)

We can separately calculate 𝑑𝑡𝑋=𝑥𝑡𝑥−1𝑑𝑡 by the power rules.

=∫0∞𝑒−𝑡𝑥𝑡𝑥−1𝑑𝑡

𝑥∫0∞𝑡𝑥−1𝑒−𝑡𝑑𝑡=𝑥Γ(𝑥)

The technique is to do the integration by parts and factor out certain quantities.

If we want to find out (b), then we can do induction once we find Γ(2)

Γ(2)=∫0∞𝑡𝑒−𝑡𝑑𝑡=∫0∞𝑡𝑑𝑒−𝑡−1=𝑡⋅𝑒−𝑡−1|𝑡=0𝑡=∞−∫𝑒−𝑡−1𝑑𝑡

=∫0∞𝑒−𝑡𝑑𝑡

What about the pdf? Given by definition 5.31

𝑓𝑎,𝜆(𝑥)=𝜆𝑎Γ(𝑎)𝑒−𝜆𝑥𝑥𝑎−1

𝑋∼Γ(𝑎,𝜆)

It has a lot of weird special cases.

Proposition 5.32 shows that this is a true density function. We will do integration from 0 to the infinity of this density

∫0∞𝜆𝑎Γ(𝑎)𝑒−𝜆𝑥𝑥𝑎−1𝑑𝑥

=𝜆𝑎Γ(𝑎)∫0∞(1𝜆𝑦)𝑎−1⋅𝑒−𝑦1𝜆𝑑𝑦

=𝜆𝑎Γ(𝑎)∫0∞1𝜆𝑎−1⋅1𝜆⋅𝑒−𝑦1𝜆𝑑𝑦

After we cancel out the 𝜆

=1Γ(𝑎)∫0∞𝑦𝑎−1𝑒−𝑦𝑑𝑦

which is just the definition of gamma

=Γ(𝑎)Γ(𝑎)=1

Proposition 5.33 says for E and V that

𝔼[𝑋]=𝑎𝜆

𝕍[𝑋]=𝑎𝜆2

We know from the definition of expectation and combining similar terms that

𝔼[𝑋]=∫0∞𝑥⋅𝜆𝑎Γ(𝑎)𝑥𝑎−1𝑒−𝜆𝑥𝑑𝑥=𝜆𝑎Γ(𝑎)∫0∞𝑥𝑎𝑒−𝜆𝑥𝑑𝑥

Let’s do a change of variable to 𝑦=𝜆𝑥 =𝜆𝑎Γ(𝑎)⋅∫0∞(𝑦𝜆)𝑎⋅𝑒−𝑦⋅1𝜆𝑑𝑦

Taking this out, I get

𝜆𝑎Γ(𝑎)⋅1𝜆𝑎1𝜆∫0∞𝑦𝑎𝑒−𝑦𝑑𝑦

Here we can use the result from before of the definition of gamma to substitute the integral.

=𝑎𝜆

=𝑎𝜆

To get variance, we have

𝔼[𝑋2]=𝜆𝑎Γ(𝑎)⋅∫0∞𝑥𝑎+1𝑒−𝜆𝑥𝑑𝑥

Change of variable…

=𝜆𝑎Γ(𝑎)⋅∫0∞(𝑦𝜆)𝑎+1𝑒𝑦𝑑𝑦

=𝑎(𝑎+1)𝜆2

𝕍[𝑋]=𝑎𝜆2

I can sequentially use the value of a to get this result.

Finally, proposition 5.35 shows us that for 𝑋∼𝑁(0,1), then 𝑋2∼Γ(12,12)

Our proof is that

𝐹𝑋2(𝑡)=ℙ(𝑋2≤𝑡)=ℙ(−𝑡≤𝑋≤𝑡)

=ℙ(𝑋≤𝑡)−ℙ(𝑋≤−𝑡)

x is a normal density, so we can write this as


now we’re going to differentiate on both sides

𝑓𝑋2(𝑡)=𝑑𝑑𝑡𝐹𝑋2(𝑡). The source note starts the derivative calculation here; the remaining algebra is not recorded.

These two minus signs cancel out and you get both

And the two will cancel out and you can put this downstairs

Then you can rewrite this exactly as Γ(12,12)

𝑓12,12(𝑥)=(12)12Γ(12)⋅𝑒−12𝑥𝑥−12

And then we can know from our definition that

12Γ(12)=12𝜋

Using the progressive values of gamma makes it easier to do calculations with them.