Lectures on Mathematical Statistics-R. C. Sickles Revised September, 1999

September 21, 1999

1. Special Distributions

1.1. Discrete Distributions.

Binomial Distribution:. The binomial distribution is denoted by B (n, p), where (n, p) are parameters. The binomial distribution tells us the probability of i successes (and thus n−i failures) in n independent trials, each of which has the same probability of success.The random variable X takes on possible values of 0, 2, ..., n, such that

P (X = i) = P (i) = ( \begin{matrix}n \\ i \end{matrix} ) p^{i} (1 − p)^{n−i}, where ( \begin{matrix}n \\ i \end{matrix} ) = \frac{n!}{i! (n − i)!}

Example 1. Example: n: the number of coin tosses (trials) with two possible outcomes on each toss (trial) and P = probability of success in a given toss.

Example 2. What is the probability of 2 heads in three tosses?

Remark 1. X ∼ B (n, p) .Let X = ∑_{i=1}^{n} X_{i}, where

X_{i} = \{ \begin{matrix}1 & if success on ith trial \\ 0 & if failure on ith trial \end{matrix} \}

Then X is the total number of successes.

Theorem 3. The mgf for the binomial distribution is

M_{x} (t) = [1 + p (e^{t} − 1)]^{n}

Proof:

M_{x} (t) = ∑_{i=0}^{n} e^{it} ( \begin{matrix}n \\ i \end{matrix} ) p^{i} (1 − p)^{n−i}

= ∑_{i=0}^{n} e^{it} ( \begin{matrix}n \\ i \end{matrix} ) (pe^{t})^{i} (1 − p)^{n−i}

= [pe^{t} + (1 − p)]^{n} = [1 + p (e^{t} − 1)]^{n} Exercise 1.1. Calculate the mean and variance of the binomial distribution

\frac{∂M_{x} (t)}{∂t} = npe^{t} [1 + p (e^{t} − 1)]^{n−1}

\frac{∂^{2}M_{x} (t)}{∂t^{2}} = npe^{t} [1 + p (e^{t} − 1)]^{n−1} + n (n − 1) p^{2}e^{2t} [1 + p (e^{t} − 1)]^{n−2}

Thus

E (X) = \frac{∂M_{x} (t)}{∂t} | t = 0 = np

E (X^{2}) = \frac{∂^{2}M_{x} (t)}{∂t^{2}} | t = 0 = np + n (n − 1) p^{2}

V (X) = E (X^{2}) − [E (X)]^{2} = np (1 − p)

Multinomial Distribution. An example is rolling a die with six outcomes n times where each roll is independent. We have n successive trials with k outcomes at each trials. If a discrete distribution has a set of possible values with the corresponding probabilities:

\begin{matrix}x_{1} & x_{2} & ⋯ & x_{k} \\ p_{1} & p_{2} & ⋯ & p_{k} \end{matrix}

such that

∑_{i=1}^{k} x_{i} = n and ∑_{i=1}^{k} p_{i} = 1

Then

P (x_{1}, x_{2}, ..., x_{k}) = \frac{n!}{x_{1}!x_{2}! ⋯ x_{k}!}p_{1}^{x_{1}}p_{2}^{x_{2}} ⋯ p_{k}^{x_{k}}

where n, p_{i} (i = 1, ..., k) and k are parameters.

Remark 2. For k = 2, the multinomial distribution collapses to the binomial distribution since x_{2} = n − x_{1} and 1 − p_{1} = p_{2}:

P (x_{1}, x_{2}) = \frac{n!}{x_{1}!x_{2}!}p_{1}^{x_{1}}p_{2}^{x_{2}} = \frac{n!}{x_{1}! (n − x_{1})!}p_{1}(1 − p_{1})^{n−x_{1}}

Example 4. Example: Roll a die 5 times. What is the probability of three 2’s, one 5 and one 5 (without worrying about order)? Let X_{i} = # of i′s, i = 1, ..., 6. p_{i} = 1/6 and n = 5

P (0, 3, 0, 0, 1, 1) = \frac{5!}{0!3!0!0!1!1!} (\frac{1}{6})^{0} (\frac{1}{6})^{3} (\frac{1}{6})^{0} (\frac{1}{6})^{0} (\frac{1}{6})^{1} (\frac{1}{6})^{1}

= \frac{1}{6^{5}} × 5 × 4 = 0.03

Poisson Distribution.

Definition 5. The Poisson distribution is a discrete distribution with possible values 0, 1, 2, ..., ∞ with probabilities

P (i) = \frac{λ^{i}e^{−i}}{i!} , where λ is a parameter

Remark 3. Both the mean and the variance of the Poisson distribution are λ.

Poisson Distribution as an Approximation of the Binomial Distribution. For the binomial distribution

f (x) = ( \begin{matrix}n \\ x \end{matrix} ) p^{x}q^{n−x}

Let np = λ

f (x) = \frac{n!}{(n − x)!x!} (\frac{np}{n})^{x} (\begin{matrix}(1 − p) & n \\ (1 − p) & x \end{matrix} )

= \frac{n!}{(n − x)!x!} (\frac{λ}{n})^{x} (\begin{matrix}(1 − λ/n) & n \\ (1 − λ/n) & x \end{matrix} )

= \frac{λ^{x}}{x!} \frac{n (n − 1) ⋯ (n − x + 1)}{n^{x}} (\begin{matrix}(1 − λ/n) & n \\ (1 − λ/n) & x \end{matrix} )

For fixed x and fixed λ, if n ⟶ ∞ the above expression in the limit is:

P (x; λ) = \frac{λ^{x}e^{−λ}}{x!} for x = 0, 1, 2, ...

Example 6. Use the Poisson as an approximation to the binomial distribution. Assume 1% of all people are left handed. What is the probability that in 200 people there are exactly 2 left handed people? Using binomial distribution:

P (X = 2) = ( \begin{matrix}200 \\ 2 \end{matrix} ) (0.01)^{2} (0.99)^{198} = 0.27203

Using Poisson distribution: λ = np = 2

P (X = 2) = \frac{2^{2}e^{−2}}{2!} = 2e^{−2} = 0.27067

As another alternative, we could use normal distribution since there is a large sample.

The Poisson Distribution as a Random Process. The Poisson process can be derived not only as an approximation to the binomial distribution but also directly from a simple and useful model of random behavior of events distributed randomly over time or space. Suppose that we have a process that triggers events through time and we wish to know at some fixed but arbitrary point in time, say t = 0, the probability that the number of events that have occurred up to that time is x. We make the following assumptions:

1. The occurrence of events in the current interval of time is independent of their occurrence or non-occurrence in past intervals.

2. The probability of exactly one occurrence of the event during the short interval (t, t + △t) is b△t (we neglect higher-order terms), where b is a positive constant. If △t is doubled, the probability of one event is also doubled. Therefore, this probability depends only on the length of the interval and not on its starting point. The probability is the same for all intervals of length △t.

3. The probability of more than one event in the short interval △t is assumed to be of lower order of magnitude that △t and is neglected; that is, the probability of two or more events in the interval △t approaches 0 faster that does △t.

From these assumptions, it is clear that if we take (0, t) and (t, t + △t) as nonoverlapping intervals, x events up to time t + △t can occur in only two ways: x events during the interval (0, t) and 0 during (t, t + △t) or x − 1 during (0, t) and 1 during (t, t + △t). If we denote by f (x, t) the probability that x events occur up to time t, then we have

f (x, t + △t) = f (x, t) · (1 − b △ t) + f (x − 1, t) · b △ t

As △t → 0 the above expression defines the differential equation

lim_{△t→∞} \frac{f (x, t + △t) − f (x, t)}{△t} = \frac{df (x, t)}{dt} = −bf (x, t) + bf (x − 1, t)

In order to solve the differential equation for f (x, t), begin with x = 0. Since f (−1, t) = 0, we get

\frac{df (0, t)}{dt} = −bf (0, t)

A solution to this equation would be f (0, t) = e^{−bt}, the constant of integration being determined by the reasonable assumption that f (0, 0) = 0.

Next, put x = 1 for which

\frac{df (1, t)}{dt} = −bf (1, t) + be^{−bt}

A solution to this equation will be f (1, t) = bte^{−bt}, where the constant of integration has been determined by the assumption that f (1, 0) = 0. If we continue to repeat this argument (also assuming f (x, 0) = 0) then

f (x, t) = \frac{e^{−bt} (bt)^{x}}{x!} for x = 0, 1, 2, ...

which is the definition for the Poisson process with λ = bt and t a constant function of time.

Theorem 7. The mgf of Poisson process is

M_{x} (t) = e^{λ(e^{t}−1)}

Proof:

M_{x} (t) = ∑_{x=0}^{∞} e^{xt} \frac{λ^{x}e^{−λ}}{x!} = e^{−λ} ∑_{x=0}^{∞} \frac{(λe^{t})^{x}}{x!} = e^{λ(e^{t}−1)}

Thus

Mx^{′} (t) = λe^{t}e^{λ(e^{t}−1)}

Mx^{′′} (t) = λe^{t}e^{λ(e^{t}−1)} + λ^{2}e^{2t}e^{λ(e^{t}−1)}

So, by letting t = 0, we have

E (X) = λ, E (X^{2}) = λ + λ^{2} ⇒ σ^{2} = λ + λ^{2} − λ^{2} = λ

The Negative Binomial Distribution. In connection with repeated Bernoulli trials, we are sometimes interested in the number of the trial on which the kth success occurs. For instance, we may be interested in the probability that the tenth child exposed to a contagious disease will be the third to catch it, the probability that the fifth person to hear a rumor will be the first one to believe it, or the probability that a burglar will be caught for the second time on his eighth job.

If the kth success is to occur on the xth trial, there must be k − 1 successes on the first x − 1 trials, and the probability of this is

b (k − 1; x − 1, p) = ( \begin{matrix}x − 1 \\ k − 1 \end{matrix} ) p^{k−1} (1 − p)^{x−k}

The probability of a success on the kth trial is p, and the probability that the kth success occurs on the xth trial is, therefore,

p · b (k − 1; x − 1, p) = ( \begin{matrix}x − 1 \\ k − 1 \end{matrix} ) p^{k} (1 − p)^{x−k}

Definition 8. A random variable X has a negative binomial distribution, and it is referred to as a negative binomial random variable, if and only if its probability distribution is given by

b^{∗} (x; k, p) = ( \begin{matrix}x − 1 \\ k − 1 \end{matrix} ) p^{k} (1 − p)^{x−k}

for x = k, k + 1, k + 2, ...

Thus, the number of the trial on which the kth success occurs is a random variable having a negative binomial distribution with the parameters k and p. The name “negative binomial distribution” derives from the fact that the values of b^{∗} (x; k, p) for x = k, k + 1, k + 2, ..., are the successive terms of the binomial expansion of (1/p − (1 − p) /p)^{−k}. In the literature of statistics, negative binomial distribution are also referred to as binomial waiting-time distributions or as Pascal distributions.

Example 9. If the probability is 0.4 that a child exposed to a certain contagious disease will catch it, what is the probability that the tenth child exposed to the disease will be the third to catch it? Substituting x = 10, k = 3, and θ = 0.4 into the formula for the negative binomial distribution, we get

b^{∗} (10; 3, 0.4) = ( \begin{matrix}9 \\ 2 \end{matrix} ) 0.4^{3} (0.6)^{7} = 0.0645

Remark 4. When a table of binomial probabilities is available, the determination of negative binomial probabilities can generally be simplified by making use of the following identity

Theorem 10.

b^{∗} (x; k, p) = \frac{k}{x} · b (x; k, p)

Theorem 11. The mean and variance of the negative binomial distribution are

μ = \frac{k}{p} and σ^{2} = \frac{k}{p} (\frac{1}{p} − 1)

Geometric Distribution. Since the negative binomial distribution with k = 1 has many important applications, it is given a special name; it is called the geometric distribution.

Definition 12. A random variable x has a geometric distribution if only if its probability distribution is given by

g (x; p) = p (1 − p)^{x−1} for x = 1, 2, 3, ...

Example 13. If the probability is 0.75 that an application for a driver’s licence will pass the road test on any given try, what is the probability that an applicant will finally pass the test on the fourth try. Substitute x = 4 and p = 0.75 into the formula for the geometric distribution, we get

g (4; 0.75) = 0.75 (1 − 0.75)^{4−1} = 0.0117

Of course, this result is based on the assumption that the trials are all independent, and there may be some question here about its validity.

The Hypergeometric Distribution. Before we define the hypergeometric distribution we will review sampling with and without replacement to illustrate the multiplication rules for independent and dependent events. To obtain a formula analogous to that of the binomial distribution which applies to sampling without replacement, in which case the trials are not independent, consider a set of N elements of which k are looked upon as successes and the other N − k as failures. As with the binomial distribution, we are interested in the probability of getting x successes in n trials, but now we are choosing, without replacement, n of the N elements contained in the set.

Remark 5. There are ( \begin{matrix}k \\ x \end{matrix} ) ways of choosing x of the k successes and ( \begin{matrix}N − k \\ n − x \end{matrix} )

ways of choosing n − x of the N − k failures, and , hence, ( \begin{matrix}k \\ x \end{matrix} ) ( \begin{matrix}N − k \\ n − x \end{matrix} ) ways

of choosing x successes and n − x failures. Since there are ( \begin{matrix}N \\ n \end{matrix} ) ways of choosing

n of the N elements in the set and we shall assume that they are all equally likely (which is what we mean when we say that the selection is random), it follows that

the probability of “x successes in n trials” is ( \begin{matrix}k \\ x \end{matrix} ) ( \begin{matrix}N − k \\ n − x \end{matrix} ) /( \begin{matrix}N \\ n \end{matrix} ) .

Definition 14. A random variable X has a hypergeometric distribution if and only if its probability distribution is given by

h (x; n, N, k) = \begin{matrix}( \begin{matrix}k \\ x \end{matrix} ) & ( \begin{matrix}N − k \\ n − x \end{matrix} ) \\ ( & \begin{matrix}N \\ n \end{matrix} & ) \end{matrix}

for x = 0, 1, 2, ..., n, x ≤ k and n − x ≤ N − k.

Thus, for sampling without replacement, the number of successes in n trials is a random variable having a hypergeometric distribution with the parameters n, N and k.

Theorem 15. The mean and the variance of the hypergeometric distribution are

μ = \frac{nk}{N} and σ^{2} = \frac{nk (N − k) (N − n)}{N^{2} (N − 1)}

Proof (mean): To determine the mean, directly evaluate the sum

μ = ∑_{x=0}^{n} x · \begin{matrix}( \begin{matrix}k \\ x \end{matrix} ) & ( \begin{matrix}N − k \\ n − x \end{matrix} ) \\ ( & \begin{matrix}N \\ n \end{matrix} & ) \end{matrix}

= ∑_{x=0}^{n} \frac{k!}{(x − 1)! (k − x)!} · \begin{matrix}( & \begin{matrix}N − k \\ n − x \end{matrix} & ) \\ ( & \begin{matrix}N \\ n \end{matrix} & ) \end{matrix} = \frac{nk}{N}

Example 16. There are 120 job applicants and 80 of them are qualified. If 5 of the applicants are randomly selected for “in-depth” interviewer, what is the probability that only 2 of the 5 will be qualified?

h (2; 5, 120, 80) = \begin{matrix}( \begin{matrix}80 \\ 2 \end{matrix} ) & ( \begin{matrix}40 \\ 3 \end{matrix} ) \\ ( & {120 \atop 5} & ) \end{matrix} = 0.164

What if we used the binomial as an approximation?

b (2; 5, \frac{120}{180}) = ( \begin{matrix}5 \\ 2 \end{matrix} ) (\frac{2}{3})^{2} (\frac{1}{3})^{3} = 0.165 ≈ h (2; 5, 120, 80)

1.2. Continuous Distribution.

Uniform Distribution.

f (x) = \{ \begin{matrix}\frac{1}{β−α} & α < x < β \\ 0 \end{matrix}

μ = \frac{α + β}{2} and σ^{2} = \frac{1}{12} (β − α)^{2}

Gamma, Exponential and Chi-Squared Distribution.

1. The gamma distribution:

f (x) = \{ \begin{matrix}kx^{α−1}e^{−x/β} & x > 0 \\ 0 & otherwise \end{matrix}

where α > 0, β > 0 and k is chosen so that

∫_{0}^{∞} kx^{α−1}e^{−x/β}dx = kβ^{α} ∫_{0}^{∞} y^{α−1}e^{−y}dy = kβ^{α}Γ (α) = 1

i.e.

k = \frac{1}{β^{α}Γ (α)}

2. The exponential distribution is parameterization of the gamma with α = 1, β = θ :

f (x) = \{ \begin{matrix}\frac{1}{θ}e^{−x/θ} & x > 0 \\ 0 & otherwise \end{matrix}

3. The chi-squared distribution is a parametrization of the gamma with α = v/2, β = 2 :

f (x) = \{ \begin{matrix}\frac{1}{2^{v/2}Γ(v/2)}x^{\frac{v−2}{2}} e^{−x/2} & x > 0 \\ 0 & otherwise \end{matrix}

Moments of gamma distribution

M_{x} (t) = ∫_{0}^{∞} e^{tx} \frac{1}{β^{α}Γ (α)}x^{α−1}e^{−x/β}dx

= \frac{1}{Γ (α)} ∫_{0}^{∞} y^{α−1}e^{−y(1−βt)}dy = \frac{1}{(1 − βt)^{α}}

Thus

μr^{′} = \frac{β^{r}Γ (α + r)}{Γ (α)} ⇒ μ = αβ and σ^{2} = αβ^{2}

Special cases: (a) Exponential μ = θ and σ^{2} = θ^{2}; (b) Chi-Squared μ = v and σ^{2} = 2v.

Beta Distribution. The uniform density f (x) = 1 for 0 < x < 1 and f (x) = 0 elsewhere is a special case of the beta distribution, which is defined in the following way:

Definition 17. A random variable X has a beta distribution, and it is referred to as a beta random variable, if and only if its probability density is given by

f (x) = \frac{Γ (α + β)}{Γ (α) · Γ (β)}x^{α−1} (1 − x)^{β−1} for 0 < x < 1 and f (x) = 0 elsewhere

where α > 0 and β > 0.

In recent years, the beta distribution has found important applications in Bayesian inference, where parameters are looked upon as random variables, and there is a need for a fairly “flexible” probability density for the parameter θ of the binomial distribution, which takes on non-zero values only on the interval from 0 to 1. By “flexible” we mean that the probability density can take on a great variety of different shapes.

We won’t prove here that the total area under the curve of the beta distribution is equal to 1, but in the proof of the theorem which follows we shall make use of the fact that

∫_{0}^{1} \frac{Γ (α + β)}{Γ (α) · Γ (β)}x^{α−1} (1 − x)^{β−1} dx = 1

and, hence, that

∫_{0}^{1} x^{α−1} (1 − x)^{β−1} dx = \frac{Γ (α) · Γ (β)}{Γ (α + β)}

This integral defines the beta function, whose values are denoted B (α, β); in other words, B (α, β) = \frac{Γ(α)·Γ(β)}{Γ(α+β)}.

Theorem 18. The mean and the variance of the beta distribution are given by

μ = \frac{α}{α + β} and σ^{2} = \frac{αβ}{(α + β)^{2} (α + β + 1)}

Proof: By definition,

μ = ∫_{0}^{1} \frac{Γ (α + β)}{Γ (α) · Γ (β)}x · x^{α−1} (1 − x)^{β−1} dx

= \frac{Γ (α + β)}{Γ (α) · Γ (β)} · \frac{Γ (α + 1) · Γ (β)}{Γ (α + β + 1)} = \frac{α}{α + β}

where we recognized the integral as B (α + 1, β) and made use of the fact that Γ (α + 1) = αΓ (α) and Γ (α + β + 1) = (α + β) Γ (α + β).

Similar steps yield

μ2^{′} = \frac{(α + 1) α}{(α + β + 1) (α + β)}

and it follows that

σ^{2} = \frac{(α + 1) α}{(α + β + 1) (α + β)} = (\frac{α}{α + β} )^{2}

= \frac{αβ}{(α + β)^{2} (α + β + 1)}

The Normal Distribution.

Definition 19. A random r.v. X has a normal distribution if its density is

f (x) = \frac{1}{σ\sqrt{2π}} exp \{−\frac{1}{2σ^{2}} (x − μ)^{2}\}

where μ and σ are parameters and σ > 0. We denote X ∼ N (μ, σ^{2}).

Facts about the normal distribution:

1. It is a distribution f (x) ≥ 0 and ∫_{−∞}^{∞} f (x) dx = 1.

2. μ = E (X), σ^{2} = var (X).

3. It is symmetric.

4. The mgf of the normal distribution is:

M_{x} (t) = ∫_{−∞}^{∞} e^{xt} \frac{1}{σ\sqrt{2π}} exp \{−\frac{1}{2σ^{2}} (x − μ)^{2}\} dx

= exp \{μt + \frac{1}{2}t^{2}σ^{2}\} \{\frac{1}{σ\sqrt{2π}} ∫_{−∞}^{∞} exp \{−\frac{1}{2} (\frac{x − (μ + tσ^{2})}{σ})^{2}\} dx\}

= exp \{μt + \frac{1}{2}t^{2}σ^{2}\}

Thus

Mx^{′} (t) = (μ + tσ^{2}) M_{x} (t)

Mx^{′′} (t) = [(μ + tσ^{2})^{2} + σ^{2}]M_{x} (t)

⇒

E (X) = μ and var (X) = σ^{2}

5. Any linear combination of normal r.v.’s has a normal distribution. This is true whether or not they are independent.

Corollary 20. If X ∼ N (μ, σ^{2}), then X^{∗} = (X − μ) /σ ∼ N (0, 1).

Since the value for F (x) are tabulated for X ∼ N (0, 1), it is useful to convert r.v.’s to this standardized values. Note that X^{∗} has mean 0, variance 1 no matter what the distribution of X is.

Example 21. X ∼ N (100, 10^{2}). Find P (95 ≤ X ≤ 110). P (95 ≤ X ≤ 110) = P (\frac{95−100}{10} ≤ \frac{X−100}{10} ≤ \frac{110−100}{10})

= P (−0.5 ≤ X^{∗} ≤ 1) = P (X^{∗} ≤ 1) − P (X^{∗} ≤ 0.5) = 0.5328

Fact 1. If X ∼ N (0, 1), then X^{2} ∼ χ_{1}^{2}.

Fact 2. If X ∼ χ_{r}^{2} and Y ∼ χ_{s}^{2} and if X and Y are independent, then

X + Y ∼ χ_{r+s}^{2}

Note: for normal distribution a linear combination of normal random variable has a normal distribution even if the r.v.’s are not independent. However for Chi-squared, independence is required.

Fact 3. If X_{1}, X_{2}, ⋯, X_{n} are iid N (0, 1), then

∑_{i=1}^{n} X_{i}^{2} ∼ χ_{n}^{2}

If X̄ = \frac{1}{n} ∑_{i=1}^{n} X_{i}, then

∑_{i=1}^{n} (X_{i} − X̄)^{2} ∼ χ_{n−1}^{2}

i.e. we loose a degree of freedom when we use X̄, because if we know X_{1}, X_{2}, ⋯, X_{n−1} and X̄, X_{n} will be known.

Student’s-t Distribution:. The t-distribution is denoted by t_{n}, where n is the number of degree of freedom. t-distribution. It is symmetric about 0 and σ^{2} = \frac{n}{n−1} > 1.

Fact 1. As n → ∞, t_{n} →^{d} N (0, 1). So for big values of n, we use the normal distribution instead.

Fact 2. If X ∼ N (0, 1), Y ∼ χ_{n}^{2} and if X and Y are independent, then

\frac{X}{\sqrt{Y/n}} ∼ t_{n}

F Distribution.. The F distribution is denoted by F_{n_{1},n_{2}}, where n_{1} and n_{2} are two different degrees of freedom. The density graph of F distribution is very similar to the Chi-Squared distribution density graph. It is defined only for the positive part of the real line and Skewed to the right.

Fact 1. If X ∼ χ_{n_{1}}^{2} and Y ∼ χ_{n_{2}}^{2} and if X and Y are independent, then

\frac{X/n_{1}}{Y/n_{2}} ∼ F_{n_{1},n_{2}}

n_{1}: the numerator degree of freedom. n_{2}: the denominator degree of freedom.

Theorem 22.

\frac{X̄ − μ}{\sqrt{S^{2}/n}} ∼ t_{N−1}

Proof:

\frac{X̄ − μ}{\sqrt{S^{2}/n}} = \frac{\frac{X̄−μ}{σ/\sqrt{n}}}{\sqrt{\frac{S^{2}/n}{σ^{2}/n}}} = \frac{N (0, 1)}{\sqrt{S^{2}/σ^{2}}}

= \frac{N (0, 1)}{\sqrt{(∑ [(X_{i} − X̄) /σ]^{2}) / (N − 1)}}

= \frac{N (0, 1)}{\sqrt{χ_{N−1}^{2}/ (N − 1)}} ∼ t_{N−1}

2. Probability Transformation

Problem: Given the density f (x_{1}, x_{2}, ⋯, x_{N}) of (X_{1}, X_{2}, ⋯, X_{N}), what is the density g (y) of Y = w (X_{1}, X_{2}, ⋯, X_{N})?

Usually we use the following three techniques to find g (y).

(1) Distribution function technique.

(2) Transformation of variables.

(3) mgf.

2.1. Distribution Function Technique. If G (y) = Pr (Y ≤ y) = Pr [w (X_{1}, X_{2}, ⋯, X_{N}) ≤ y], then g (y) = dG (y) /dy.

Ex 1.

f (x) = \{ \begin{matrix}6x (1 − x) & 0 < x < 1 \\ 0 & otherwise \end{matrix}

If y = x^{3},

G (y) = Pr (Y ≤ y) = Pr (X^{3} ≤ y) = Pr (X ≤ y^{1/3})
= ∫_{0}^{y^{1/3}} 6x (1 − x) dx = 3y^{2/3} − 2y

⇒

g (y) = \{ \begin{matrix}2 (y^{−1/3} − 1) & 0 < y < 1 \\ 0 & otherwise \end{matrix}

Ex. 2: If Y = |X|, then

g (y) = \{ \begin{matrix}f (y) + f (−y) & y > 0 \\ 0 & otherwise \end{matrix}

Proof: For y > 0, G (y) = Pr (Y ≤ y) = Pr (|X| ≤ y) = Pr (−y ≤ X ≤ y). Thus

G (y) = F (y) − F (−y)
g (y) = f (y) + f (−y) for y > 0

Ex. 3:

f (x_{1}, x_{2}) = \{ \begin{matrix}6e^{−3x_{1}−2x_{2}} & for x_{1} > 0 and x_{2} > 0 \\ 0 & otherwise \end{matrix}

Let Y = X_{1} + X_{2}, then

F (y) = ∫_{0}^{y} ∫_{0}^{y−x_{2}} 6e^{−3x_{1}−2x_{2}}dx_{1}dx_{2} = 1 + 2e^{−3y} − 3e^{−2y}
f (y) = dF (y) /dy = 6 (e^{−2y} − e^{−3y}) for y > 0

2.2. Transformation of Variable Technique.

Theorem 23. Let f (x) be the density function of random variable X. y = u (x) is monotonic in x for f (x) ≠ 0 and is differentiable. Then there is a unique one-to-one mapping from y to x given by x = w (y) and

g (y) = \{ \begin{matrix}f [w (y)] · |w′ (y)| & for u′ (x) ≠ 0 \\ 0 & otherwise \end{matrix}

Proof: First we assume that y = u (x) is monotonic increasing,

Pr (a < Y < b) = Pr [w (a) < X < w (b)]
= ∫_{w(a)}^{w(b)} f (x) dx = ∫_{a}^{b} f (w (y)) w′ (y) dy

Thus we get

g (y) = f (w (y)) w′ (y)

Second we assume that y = u (x) is monotonic decreasing,

Pr (a < Y < b) = Pr [w (b) < X < w (a)]
= ∫_{w(b)}^{w(a)} f (x) dx = − ∫_{a}^{b} f (w (y)) w′ (y) dy

Thus we get

g (y) = −f (w (y)) w′ (y)

But when y = u (x) is monotonic decreasing, −w′ (y) > 0. So we get

g (y) = f (w (y)) |w′ (y)|

Ex. X ∼ N (0, 1) and Z = X^{2}. In this example the transformation is not monotonic in x, we will do it is two steps.

First, if y = |x|, ⇒

g (y) = \{ \begin{matrix}\frac{2}{\sqrt{2π}}e^{− \frac{1}{2} y^{2}} & for y > 0 \\ 0 & otherwise \end{matrix}

Second, z = y^{2} and

\frac{dy}{dz} = \frac{1}{2}z^{− \frac{1}{2}}

h (z) = \frac{2}{\sqrt{2π}} e^{− \frac{1}{2} z} |\frac{1}{2}z^{\frac{1}{2}} | = \frac{1}{\sqrt{2π}} e^{− \frac{1}{2} z}z^{\frac{1}{2}} for z > 0

Note Γ (1/2) = \sqrt{π}, where

Γ (α) = ∫_{0}^{∞} y^{α−1}e^{−y}dy

Thus

h (z) = \frac{1}{2^{1/2}Γ (1/2)}e^{− \frac{1}{2} z}z^{\frac{1}{2}} for z > 0

is the density function of χ_{1}^{2}.

Ex.

f (x_{1}, x_{2}) = \{ \begin{matrix}1 & for 0 < x_{1}, x_{2} < 1 \\ 0 & otherwise \end{matrix}

y = x_{1} + x_{2}

First we find the joint density of y and x_{2}.

x_{1} = y − x_{2} and ∂x_{1}/∂y = 1

Thus

g (y, x_{2}) = 1 · |1| = \{ \begin{matrix}1 & for 0 < x_{2} < 1 and x_{2} < y < x_{2} + 1 \\ 0 & otherwise \end{matrix}

Next integrate over x_{2}.

h (y) = \{ \begin{matrix}∫_{−∞}^{0} g (y, x_{2}) dx_{2} = 0 & y ≤ 0 \\ ∫_{0}^{y} 1dx_{2} = y & 0 < y < 1 \\ ∫_{y−1}^{1} 1dx_{2} = 2 − y & 1 < y < 2 \\ ∫_{2}^{∞} g (y, x_{2}) dx_{2} = 0 & y ≥ 2 \end{matrix}

t-Distribution:

Let

T = \frac{X}{\sqrt{u/k}}

where X ∼ N (0, 1), u ∼ χ_{k}^{2} and X and u are independent. What is h (t)?

Theorem 24.

h (t) = \frac{1}{\sqrt{kπ}} · \frac{[(k − 1) /2]!}{[(k − 2) /2]!} (1 + \frac{t^{2}}{k} )^{−(k+1)/2} for − ∞ < t < ∞

Proof: Since u and X are independent, the joint distribution of them are

f (x, u) = \frac{1}{\sqrt{2π}} e^{− \frac{1}{2} x^{2}} · \frac{1}{[(k − 2) /2]!2^{k/2}} u^{(k−2)/2}e^{− \frac{1}{2} u}

Let

t = \frac{x}{\sqrt{v/k}} and v = u

Substitute x and u and then integrate out x,

\frac{∂x}{∂t} = \sqrt{\frac{v}{k}}

\frac{∂x}{∂v} = \frac{t}{2k} (\frac{v}{k} )^{−1/2}

\frac{∂u}{∂t} = 0

\frac{∂u}{∂v} = 1

Thus the Jacobian of the transformation from t and v to x and u is

| \begin{matrix}\frac{∂x}{∂t} & \frac{∂x}{∂v} \\ \frac{∂u}{∂t} & \frac{∂u}{∂v} \end{matrix} | = \sqrt{\frac{v}{k}}

So that

g (t, v) = \frac{1}{\sqrt{2π}} e^{− \frac{vt^{2}}{2k}} · \frac{1}{[(k − 2) /2]!2^{k/2}} u^{(k−2)/2}e^{− \frac{1}{2} v} · \sqrt{\frac{v}{k}}

and

h (t) = ∫_{0}^{∞} g (t, v) dv = \frac{1}{\sqrt{kπ}} · \frac{1}{[(k − 2) /2]!}A

where

A = \frac{1}{2} ∫_{0}^{∞} e^{−(1/2)(t^{2}/k+1)v} (\frac{v}{2} )^{(k−1)/2} dv

Let

α = \frac{k − 1}{2} and z = \frac{1}{2} (1 + \frac{t^{2}}{k} ) v

Then

\frac{v}{2} = (1 + \frac{t^{2}}{k} )^{−1} z and (\frac{v}{2} )^{α} = (1 + \frac{t^{2}}{k} )^{−α} z^{α}

Also

d (\frac{v}{2} ) = (1 + \frac{t^{2}}{k} )^{−1} dz

With these substitutions

A = (1 + \frac{t^{2}}{k} )^{1(k+1)/2} ∫_{0}^{∞} z^{α}e^{−z}dz

But

∫_{0}^{∞} z^{α}e^{−z}dz = Γ (α + 1) = [\frac{k − 1}{2}]!

So that

h (t) = \frac{1}{\sqrt{kπ}} · \frac{[(k − 1) /2]!}{[(k − 2) /2]!} (1 + \frac{t^{2}}{k} )^{−(k+1)/2} for − ∞ < t < ∞

Note: This density function is symmetric about t = 0 and all those odd moments, if they exist, vanish. Even moments are given by

μ2r^{′} = k^{r} \frac{Γ (r + 1/2) Γ (k/2 − r)}{Γ (1/2) Γ (k/2)}

for r = 0, 1, ⋯, m < k/2. Moments of order greater or equal to k do not exist. Thus E (T) = 0 for k > 1 and v (T) = k/ (k − 2) for k > 2.