Lectures On Mathematical Statistics-R. C. Sickles Revised September, 1999

September 21, 1999

1. Random Variables and Probability Distributions

In the previous material we have develped the basic notions of a random event and of probability and have derived the fundamental laws of the latter. We now introduce a new concept, that of a random variable, and begin a study of probability distributions. We start with random variables and probability distributions in a finite sample space and then proceed to random variables that can take on a continuum of values.

1.1. Discrete Random Variables:. In formulating the axioms of probability we defined probabilities over a field of events F which had been constructed as the set of random events for a particular experiment S. The probability function was then taken to be a real-valued function which assigned a number between 0 and 1 (including the endpoints) to each event in F. Before the experiment is performed, however, the outcome is uncertain, which means that we can view the outcome as variable. Since it is always possible to put the random events into a one-to-one correspondence with the integers, the random events can be defined as numbers, and consequently, the variable also can be viewed as numerical. Accordingly let us formulate the following general definitions:

Definition 1. A random variable is a variable which takes on its various values with certain (fixed) probabilities. It is a variable whose value is a number determined by the outcme of an experiment.

Example 2. A coin toss:

X = \{ \begin{matrix}1 & (head) \\ 0 & (tail) \end{matrix}

Then P (X = 1) = 1/2 = P (X = 0).

Definition 3. A probability distribuiton for a random variable X is a set of possble values (outcome) x_{1}, x_{2}, ..., x_{n} and corresponding probabilities P (x_{1}) , P (x_{2}) , ..., P (x_{n}). The set G whose elements are the ordered pairs (x_{i}, P (x_{i})) , i = 1, 2, ..., n, is called the probability distribution of X. The pairs satisfy the following axioms:

Axiom 1. P (x_{i}) ≥ 0, i = 1, 2, ..., n

Axiom 2. ∑_{i=1}^{n} P (x_{i}) = 1

1.2. Continuous Random Variables. A discrete random variables can assume only a finite, or a countably infinite, number of values. That is, it is restricted to an isolated set of numbers on the real line, real plane, etc. However, in many situations we encounter random variables that can assume any value within an nterval or a collection of intervals on the real line, in the real plane, etc., and we need to extend our definition to cover these situations.

Example 4. The average salary increase in some beaurocracy is 6%. Because minor variations in the productivity of workers and other factors, salary, if increased, differs. Let X be a variable that measures the deviation of a salary increase from the stated increase. It is clear that X is a random variables that can assume a contunuum of values.

With discrete random variables it is possible to have a nonzero probability associated with each admissible event, even in the case where the number of events is countably infinite, and still have the probabilities sum to 1. With a continuous random variable this is not possible.On the other hand, it is clear that all values of X within a neighborhood of a particular point will be about equally probable, and so to say that most of these points have probability 0 while a few others have nonzero probabilities does not much sense either. Therefore, it would seen that we have encountered a problem.

Before we establish a way to get around the problem it should be pointed that it is a logical rather than a practical problem. From a practical point of view, since we are limited by the accuracy of our measuring devices, a deviation can only be identified within a certain interval. Thus as a practical matter only probabilities within an interval have meaning for a continuous random variable.

Definition 5. A continuous probability distribution for a r.v. X is specified by a density function f (x) such that

f (x) ≥ 0 for all x

∫_{−∞}^{∞} f (x) dx = 1 Example 6. Consider a uniform distribution on [0, 1] defined by the following density function

f (x) = \{ \begin{matrix}1 & if x ∈ [0, 1] \\ 0 & otherwise \end{matrix} \}

In a uniform distribution, all points x ∈ [0, 1] have the same density so that, for example:

P (X ≤ 1/2) = ∫_{0}^{1/2} 1dx = 1/2

Definition 7. The cummulative distribution function (cdf) of a r.v. X with density function f (x) is

F (x) = P (X ≤ x) = ∫_{−∞}^{x} f (t) dt

For the above example

F (1/2) = P (X ≤ 1/2) = ∫_{0}^{1/2} 1dx = 1/2

Remark 1. Note that:

1. dF (x) /dx = f (x)

2. P (a < x ≤ b) = P (X ≤ b) − P (X ≤ b) = F (b) − F (a)

3. F (x) ≡ P (X ≤ x) = P (X < x)

1.3. Moments of a Random Variable.

Definition 8. Let X be a r.v. Then the expected value (or mean) of X, denoted by E (X) is

E (X) = ∑_{i=1}^{n} x_{i}P (x_{i}) , if X is discrete
E (X) = ∫_{−∞}^{∞} xf (x) dx, if X is continuous

Example 9. Roulette wins $35 with probability 1/38 or losses $1 with probability 37/38

\begin{matrix}x_{1} = 35 & P (x_{1}) = 1/38 \\ x_{2} = −1 & P (x_{2}) = 37/38 \end{matrix}

E (X) = 35 × (1/38) + (−1) × (37/38) = −2/38 = −$0.0526 Example 10. Uniform distribution

f (x) = \{ \begin{matrix}1 & if x ∈ [0, 1] \\ 0 & otherwise \end{matrix}

E (X) = ∫_{−∞}^{∞} xf (x) dx = ∫_{0}^{1} x × 1dx = \frac{1}{2}x^{2}|_{0}^{1} = \frac{1}{2}

Moments and Mechanics.

Definition 11. (Assume that an object of mass m occupies a single point. Let L be the axis.) The moment of inertia of a particle of mass m about the axis L is mr^{2}, where r is the perpendicular distance of the object from the axis.

Definition 12. If we have a system of particles m_{1}, ..., m_{n} at perpendicular distances r_{1}, ..., r_{n} from L, then the moment of inertia of the system, I, is given by

I = m_{1}r_{1}^{2} + ⋯ + m_{n}r_{n}^{2} = ∑_{i=1}^{n} m_{i}r_{i}^{2}

Remark 2. The expected value does not necessarily exist.

Example 13. Cauchy Distribution density is f (x) = \frac{1}{π(1+x^{2})}which satisfies

(1) f (x) ≥ 0

(2)

∫_{−∞}^{∞} f (x) dx = \frac{1}{π} ∫_{−∞}^{∞} \frac{1}{1 + x^{2}} dx = \frac{1}{π} arctan (x) |_{−∞}^{∞} = \frac{1}{π} (\frac{π}{2} − (−\frac{π}{2} )) = 1

(3)

E (x) = ∫_{−∞}^{∞} xf (x) dx = \frac{1}{π} ∫_{−∞}^{∞} \frac{x}{1 + x^{2}} dx = \frac{1}{2π} ln (1 + x^{2}) |_{−∞}^{∞} =?

In this case there is so much probability in the tails of the distribution that the intergral never converge.

Remark 3. Another way for the mean not to exist is for it to equal to ∞.

Example 14. Petersberg’s Paradox: Player is to receive $2^{x} if, in a series of coin flips, the first head appears on the xth flip, i.e., he/or she gets $2 if the head happens on the first flip, $4 if the first head happeds on the second time flip, etc. As it is easy to figure out f (x) = (1/2)^{x}, thus

E (x) = 2 (1/2) + 4 (1/4) + ⋯ + 2^{x} (1/2^{x}) + ⋯ = +∞

The expected value is infinite or we might say it does not exist.

Definition 15. Let X be a r.v. and g be a function. Then the expected value of g(x) is

(discrete case)

E [g (x)] = ∑_{i=1}^{n} g (x_{i}) P (x_{i})

(continuous case)

E [g (x)] = ∫_{−∞}^{∞} g (x) f (x) dx

Example 16. Toss a coin.

X = \{ \begin{matrix}1 & (head) \\ 0 & (tail) \end{matrix}
E (X^{2}) = 1^{2} (1/2) + 0^{2} (1/2) = 1/2

Theorem 17. If X is a r.v. and c is a number, then E [cX] = cE [X].

Proof: Discrete case

E [cX] = ∑_{i=1}^{n} cx_{i}P (x_{i}) = c ∑_{i=1}^{n} x_{i}P (x_{i}) = cE (X)

Corollary 18. If c is a constant, then E (c) = c.

Corollary 19. If c and d are constants, then E (c + dX) = c + dE (X).

Definition 20. Let X be a r.v. with mean E (X) = μ. Then the variance of X is var (X) = E (X − μ)^{2} , or

var (X) = ∑_{i=1}^{n}(x_{i} − μ)^{2}P (x_{i})

var (X) = ∫_{−∞}^{∞} (x − μ)^{2} f (x) dx Corollary 21.

var (X) = E (X^{2}) − μ^{2}

Proof

var (X) = ∫_{−∞}^{∞} (x − μ)^{2} f (x) dx
= ∫_{−∞}^{∞} (x^{2} − 2μx + μ^{2}) f (x) dx
= E (X^{2}) − 2μE (X) + μ^{2} = E (X^{2}) − μ^{2}

Example 22. Coin toss

X = \{ \begin{matrix}1 & (head) & P (1) = 1/2 \\ 0 & (tail) & P (0) = 1/2 \end{matrix}

var (X) = E (X − μ)^{2}

= (1 − 1/2)^{2} (1/2) + (0 − 1/2)^{2} (1/2) = 1/4 var (X) = E(X^{2}) − μ^{2} = 1/2 − 1/4 = 1/4

Example 23. Uniform distribution

f (x) = \{ \begin{matrix}1 & if x ∈ [0, 1] \\ 0 & otherwise \end{matrix}

μ = E (X) = ∫_{−∞}^{∞} xf (x) dx = ∫_{0}^{1} x × 1dx = \frac{1}{2}x^{2}|_{0}^{1} = \frac{1}{2}

E (X^{2}) = ∫_{−∞}^{∞} x^{2}f (x) dx = ∫_{0}^{1} x^{2}dx = 1/3
var (X) = E(X^{2}) − μ^{2} = 1/3 − 1/4 = 1/12

Remark 4. Facts about Variances: 1. Variance≥ 0.

2. If X is a r.v. and c is a number, then var (cX) = cvar (X).

3. Variance = 0, if and only if X is a constant.

4. Variances may not exists: If the mean does not exist, the variance does not exist; If the mean exists, the variance may or may not exist. However, with the reasonable distribution, we will use the variance exist if the mean exists.

Definition 24. The standard deviation of X is \sqrt{var (X)}.

To see how σ^{2} is the indicator of the dispersion of the distribution of a random variable let’s prove the following theorem.

Theorem 25. (Chebychev’s Inequality) Let X be a r.v. with mean μ and variance σ^{2} > 0. Then for any positive number λ

P (|X − μ| ≥ λσ) ≤ 1/λ^{2} or P (|X − μ| ≤ λσ) ≥ 1 − 1/λ^{2}

Proof:

σ^{2} = ∫_{−∞}^{∞} (x − μ)^{2} f (x) dx

= ∫_{−∞}^{μ−λσ} (x − μ)^{2} f (x) dx+ \overbrace{∫_{μ−λσ}^{μ+λσ} (x − μ)^{2} f (x) dx}^{positive} + ∫_{μ+λσ}^{∞} (x − μ)^{2} f (x) dx

≥ ∫_{−∞}^{μ−λσ} (x − μ)^{2} f (x) dx + ∫_{μ+λσ}^{∞} (x − μ)^{2} f (x) dx
≥ λ^{2}σ^{2} ∫_{−∞}^{μ−λσ} f (x) dx + λ^{2}σ^{2} ∫_{μ+λσ}^{∞} f (x) dx
= λ^{2}σ^{2}P (|X − μ| ≥ λσ)

Thus

P (|X − μ| ≥ λσ) ≤ 1/λ^{2} or P (|X − μ| ≤ λσ) ≥ 1 − 1/λ^{2}

2. Moment-Generating Functions

Definition 26. The moment generating function (mgf) (if it exists) is given by

M_{x} (t) = E (e^{tX}) = ∫ e^{tx}dF (x)

Thus when X is absolutely continuous

M_{x} (t) = E (e^{tX}) = ∫ e^{tx}f (x) dx

when X is discrete

M_{x} (t) = E (e^{tX}) = ∑_{i=i}^{n} p_{i}e^{tx}

where t is a real-valued variable and we are usually interested in values of t close to zero.

Let’s obtain properties of f (x) directly from M (t), assuming that M (t) exists. For −h < t < h, express e^{tx} as its MacLaurin series expansion

e^{tx} = 1 + tx + \frac{t^{2}x^{2}}{2!} + ⋯ + \frac{t^{r}x^{r}}{r!} + ⋯

Note: A MacLaurin series is a Taylor series expansion about 0

f (x) = ∑_{n=0}^{∞} \frac{f^{n} (a)}{n!} (x − a)^{n}, for a = 0

Thus

M_{x} (t) = ∫ f (x) dx + t ∫ xf (x) dx + \frac{t^{2}}{2!} ∫ x^{2}f (x) dx + ⋯ + \frac{t^{r}}{r!} ∫ x^{r}f (x) dx + ⋯

= 1 + tμ_{1} + \frac{t^{2}}{2!}μ_{2} + ⋯ + \frac{t^{r}}{r!}μ_{r} + ⋯

Remark 5. There are some helpful relationships involving the mfg. They are:

1. μ_{r} = d^{r}M_{x} (t) /dt^{r}|_{t=0}

2. If a and b are constant, then

M_{x+a} (t) = e^{at}M_{x} (t) M_{bx} (t) = M_{x} (bt) M_{(x+a)/b} (t) = e^{at/b}M_{x} (t/b)

Example 27. Assume p (x) = \frac{1}{8} \binom{3}{x} for x ∈ \{0, 1, 2, 3\} and 0 otherwise. What is μ_{1} and μ_{2}?

M_{x} (t) = E (e^{tX}) = \frac{1}{8} ∑_{x=0}^{3} e^{tx} \binom{3}{x} = \frac{1}{8} (1 + 3e^{t} + 3e^{2t} + e^{3t}) = \frac{1}{8} (1 + e^{t})^{3}

Therefore

μ_{1} = Mx^{′} (0) = \frac{3}{8} (1 + e^{0})^{2} e^{0} = \frac{12}{8}

μ_{2} = Mx^{′′} (0) = \frac{6}{8} (1 + e^{t}) e^{2t} + \frac{3}{8} (1 + e^{t})^{2} e^{t}|_{t=0} = 3

Lectures On Mathematical Statistics-R. C. Sickles Revised September, 1999 9

Example 28.

f (x) = \{ \begin{matrix}e^{−x} & for x > 0 \\ 0 & 0 otherwise \end{matrix}

This is a special case of exponential distribution f(x) = \frac{1}{θ}e^{−x/θ} for x > 0.

M_{x} (t) = ∫_{0}^{∞} e^{tx}e^{−x}dx = \frac{e^{x(t−1)}}{t − 1} |_{x=0}^{x=∞} = \frac{1}{1 − t}

= 1 + t + t^{2} + t^{3} + ⋯ + t^{r} + ⋯ for |t| < 1

Thus μ_{r} = r!. Example 29.

f (x) = \{ \begin{matrix}1 & for x ∈ [0, 1] \\ 0 & 0 otherwise \end{matrix}

M_{x} (t) = ∫_{0}^{1} e^{tx}dx = \frac{1}{t} [e^{t} − 1]

Mx^{′} (t) = \frac{1}{t^{2}} [te^{t} − e^{t} + 1]

Use L’Hopital’s Rule we have

\frac{1}{t^{2}} [te^{t} − e^{t} + 1] |_{t=0} = \frac{te^{t}}{2t} |_{t=0} = \frac{1}{2}

Thus μ_{1} = 1/2. Use the same method, we can obtain μ_{2} = 1/3. Higher Order Moments:

\begin{aligned}μ_{r} & = & E (X^{r}) & mean \\ μr^{′} & = & E(X − μ)^{r} & variance \\ μ3^{′} & = & E (X − μ)^{3} & skewness \\ μ4^{′} & = & E (X − μ)^{4} & kurtosis \end{aligned}

3. Applications of Moment-generating Functions Techique General problem: Given the joint density f (x_{1}, x_{2}, ⋯, x_{n}) and functions Y_{i} = g_{i} (x_{1}, x_{2}, ⋯, x_{n}), i = 1, 2, .., k., what is the joint distribution of (Y_{1}, Y_{2}, ⋯, Y_{k})?

Theorem 30. If the mgf exists, it is unique and uniquely determines the distribution function.

Remark 6. Since the joint mgf of (Y_{1}, Y_{2}, ⋯, Y_{k}) is

m_{Y_{1},Y_{2},···,Y_{k}} (t_{1}, t_{2}, ⋯, t_{k}) = E (exp {t_{1}Y_{1} + ⋯ + t_{k}Y_{k}})

= ∫ ⋯ ∫ exp {t_{1}g_{1} (x_{1}, ⋯, x_{n}) + ⋯ + t_{k}g_{k} (x_{1}, ⋯, x_{n})} f (x_{1}, ⋯, x_{n}) dx_{1} ⋯ dx_{n}

the idea is to perform the integration and hope that you can recognize the resulting mgf as the mgf of some known distribution. This technique is of limited use when k > 1 since we can recognize only a few joint mgf. But when k = 1 we can oftentimes recognize it.

Example 31. X ∼ N (0, 1), Y = X^{2}. What is g (y)?

m_{Y} (t) = ∫_{−∞}^{∞} e^{tx^{2}} \frac{1}{\sqrt{2π}} e^{− \frac{1}{2} x^{2}}dx

= \frac{1}{\sqrt{2π}} ∫_{−∞}^{∞} e^{− \frac{1}{2} x^{2}(1−2t)}dx

= \frac{1}{\sqrt{2π}} \begin{matrix}(1 − 2t) & −1/2 \\ (1 − 2t) & −1/2 \end{matrix} ∫_{−∞}^{∞} e^{− \frac{1}{2} x^{2}(1−2t)}dx

= (1 − 2t)^{− \frac{1}{2}} \frac{1}{\sqrt{2π}} ∫_{−∞}^{∞} e^{− \frac{1}{2} y^{2}}dy

= (1 − 2t)^{− \frac{1}{2}} = ( \frac{\frac{1}{2}}{\frac{1}{2} − t})^{\frac{1}{2}}

Which is the mgf of a Gamma distribution with parameters r = 1/2, λ = 1/2.

Theorem 32. If X_{1}, X_{2}, ..., X_{n} are independent r.v. and if the mgf of each exists for all −h < t < h for some h > 0 and Y = ∑_{i=1}^{n} X_{i}, then

m_{Y} (t) = E [exp (∑_{i=1}^{n} X_{i}t)] = ∏_{i=1}^{n} m_{X_{i}} (t)

Proof: Trivial.

Example 33. Assume X_{1}, X_{2}, ..., X_{n} are independent r.v’s. with X_{i} ∼ N (μ_{i}, σ_{i}^{2}). Then a_{i}X_{i} ∼ N (a_{i}μ_{i}, a_{i}^{2}σ_{i}^{2}).

m_{a_{i}X_{i}} (t) = exp (a_{i}μ_{i}t + \frac{1}{2}a_{i}^{2}σ_{i}^{2}t^{2})

m_{∑ a_{i}X_{i}} (t) = ∏_{i} exp (a_{i}μ_{i}t + \frac{1}{2}a_{i}^{2}σ_{i}^{2}t^{2})

= exp ((∑_{i} a_{i}μ_{i})t + \frac{1}{2}(∑_{i} a_{i}^{2}σ_{i}^{2})t^{2})

which is the mgf of a normal r.v. Thus

∑_{i} a_{i}X_{i} ∼ N (∑_{i} a_{i}μ_{i}, ∑_{i} a_{i}^{2}σ_{i}^{2})

4. Characteristic Functions and Related Theorems

It was noted in the previous section that the moment-generating function of a distribution does not always exist. In case such as this it is necessary to with the characteristic function. This section defines the characteristic function of a distribution and shows how it can be used to characterize the distribution. In addition, we shall prove several theorems which are of major importance to the development of the theory of statistics.

Although the discussion in this section assumes that the reader has had some contact with the theory of complex numbers, the actual demands in this regard are minimal. The purpose here isnot to elucidate the theory of characteristic functions, for this is a task for advanced courses, but rather only to lay the groundwork for the last theorem of this section, which establishes the uniqueness of the momentgenerating function in situations where the function exists. Our discussion here will be restricted to distributions for which the density and distribution functions are related by f (x) = F′ (x).

Definition 34. Let X be a random variable with density function f (x) and distribution function F (x). The characteristic function ϕ (t) of the random variable X is then defined as

ϕ (t) = E (e^{itX}) = ∫_{−∞}^{∞} e^{itx}f (x) dx

where i is the imaginary unit, i = \sqrt{−1}, and t is real. Alternatively, ϕ (t) is frequently referred to as the characteristic function corresponding to the distribution F (x).

Remark 7. Since, by definition

e^{itx} = cos tx + i sin tx,

it follows that

|ϕ (t)| ≤ ∫_{−∞}^{∞} |e^{itx}|f (x) dx = 1

This implies that the integral in the definition of ϕ (t) will exist for any F (x) for all t, and hence that, unlike the moment-generating function, the characteristic function of a random variable always exists.

Let us now assume that the rth moment of X exists. Differentiation of ϕ (t) with respect to t h times the yields

ϕ^{h} (t) = i^{h} ∫_{−∞}^{∞} x^{h}e^{itx}f (x) dx

which, upon valuation at t = 0, becomes

ϕ^{h} (0) = i^{h} ∫_{−∞}^{∞} x^{h}f (x) dx = i^{h}μh^{′}, for 0 ≤ h ≤ r

or

μh^{′} = \frac{ϕ^{h} (0)}{i^{h}}

This establishes that the characteristic function can be used as a moment-generating function when the mgf exists.

Situations often arise in probability theory and its applications, especially to sampling theory, in which it is relatively easy to find the characteristic function of a function of the components of a multidimensional random variable, paticularly a linear function. In these situations, the question then arises of how to find the distribution of the random variable from its characteristic function. In many cases, the answer to this question is answered by the follwing theorem due to Paul Levy.

Theorem 35. (Inversion Theorem) Let X be a random variable with characteristic function ϕ (t) and distribution function F (x). Assume that F (x) is continuous at x = x′ ± δ, δ > 0. Then we have

F (x′ + δ) − F (x′ − δ) = lim_{A→∞} \frac{1}{π} ∫_{−A}^{A} \frac{sin δt}{t}e^{−itx}ϕ (t) dt

Moreover, if

∫_{−∞}^{∞} ϕ (t) dt < ∞,

a density function f (x) exists at x = x′ and

f (x′) = \frac{1}{2π} ∫_{−∞}^{∞} e^{−itx}ϕ (t) dt

Proof: See Wilks (1962, pp. 116-117).

The importance of the inversion theorem can hardly be overemphasized, for as an immediate consequence of it, we have the following.

Theorem 36. A necessary and sufficient condition for two random variables X_{1} and X_{2} to have identical distribution functions is for their characteristic functions to be identical.

Proof: This can be seen from the Inversion Theorem above and the definition of the characteristic function.

Next, let us note that if t in ϕ (t) is replaced by t′ = t/i, then ϕ (t′) is the moment-generating function for the random variable X. Consequently, in situations where the moment- generation function M (t) exists, we have as a special cas of the above theorem.

Theorem 37. Let X_{1} and X_{2} be two random variables with moment-generating functions M_{1} (t) and M_{2} (t). Then a necesary and sufficient condition for X_{1} and X_{2} to have identical distribution functions is for M_{1} (t) and M_{2} (t) to be identical.

The final result on this topic deals with the uniqueness of the moments of a distribution.

Theorem 38. Let F (x) be a distribution function with moments μr^{′}, r = 0, 1, ..., all of which are finite. If the series

∑_{r=0}^{∞} \frac{μr^{′}}{r!} c^{r}

is absolutely convergent for some c > 0, then F (x) is the only distribution function having these moments.

Proof: See Wilks (1962, pp. 125-126).

Corollary 39. If X is a bounded random variable, then its distribution function F (x) is uniquely determined by its moments μr^{′}, r = 0, 1, ....

Proof: Since X is bounded, there therefore exist finite numbers a, b, a > b, such that F (a) = 0 and F (b) = 0. Let M denote the largest of |a| and |b|. The we have

|μr^{′}| ≤ ∫_{a}^{b} |x|^{r} f (x) dx = vr^{′} ≤ M^{r}

and

∑_{r=0}^{∞} |\frac{μr^{′}}{r!} c^{r}| ≤ ∑_{r=0}^{∞} \frac{vr^{′}}{r!} c^{r} ≤ ∑_{r=0}^{∞} \frac{|Mc|^{r}}{r!} = exp (|Mc|)

However, exp (|Mc|) is finite for all c. The sufficient condition in the above theorem is accordingly satisfied, thus establishing the result.

4.1. More on Characteristic Functions.

Remark 8. Every distribution function has a unique c.f.; and to each c.f. there corresponds a unique d.f. If X has a distribution function with c.f. ϕ (t), then if E (X) and E (X^{2}) exist, they are given by iE (X) = ϕ′ (0), and i^{2}E (X^{2}) = ϕ′′ (0). Moreover, if the moment generatin function exist, then ϕ (t) = M (it).

Remark 9. For those familiar with Laplace and Fourier transformation you will note the similarity between m.g.t and c.f. It is the uniqueness of their transformation that allows us to assure the uniqueness of m.g.f. and the c.f.

Example 40. Example 41. The Cauchy distribution is a special form of the Pearson Type VII distribution.

f (x) = \frac{1}{πλ} [1 + \{(x − θ) /λ\}^{2}]^{−1} for λ > 0

F (x) = \frac{1}{2} + π^{−1} tan^{−1} [(x − θ) /λ]

The parameters θ and λ are location and scale parameters. The distribution is symmetrically about x = θ. The median is θ and the upper and lower quantiles are θ ± λ. The points of inflection are θ ± λ/3. The values of the distribution at the points of inflection are 0.273 and 0.727 as compared to 0.159 and 0.841 for the normal. The standardized form is obtained by putting θ = 0; λ = 1. The tables are comparisons between Cauchy and normal distributions.The Cauchy distribution is in standard form; the normal distribution has expected value zero and standard deviation (0.67445)^{−1} = 1.4827. The two distributions have the same median (x = 0) and upper and ower quantles (x = ±1). As both distributions are symetrical about x = 0,

x Pr [X ≤ x]  x Pr [X ≤ x]
Cauchy Normal  Cauchy Normal
0.0 0.5000  0.5000 2.2 0.8642  0.9861
0.2 0.5628  0.5793 2.4 0.8743  0.9918
0.4 0.6211  0.6554 2.6 0.8831  0.9953
0.6 0.6720  0.7257 2.8 0.8908  0.9974
0.8 0.7148  0.7881 3.0 0.8976  0.9987
1.0 0.7500  0.8413 3.2 0.9036  0.9993
1.2 0.7789  0.8849 3.4 0.9089  0.9997
1.4 0.8026  0.9192 3.6 0.9138  0.9924
1.6 0.8222  0.9452 3.8 0.9181  0.9948
1.8 0.8386  0.9641 4.0 0.9220  0.9965
2.0 0.8524  0.9772

Table 1: Comparison of Cauchy and Normal Distribution

there is no need to tabulate negative values of x.The characteristic functions of the Cauchy and normal distributions are, respectively:

ϕ (t) = exp [it − |t| λ]

ϕ (t) = exp [−\frac{1}{2}t^{2}]

5. Random Variables in n-Dimensions

Thus far our discussion has been entirely in terms of probability distributions for a single random variable. We now extend the basic notions of probability to a random variable in several dimensions.

5.1. Two Dimensional Case. Although we did not take note of it at the time, we have already been in contact with the joint distribution of two random variables, namely the experiment of tossing two dice. The two random variables are, respectively, the faces of the two dice. The joint distribution of the two random variables is given by 36 possible outcomes with their associated probabilities. If we denote the face of the first die by X and that of the second by Y, then the density function for the joint occurrence of X = x and Y = y (assuming that the dice are fair) is:

P (X = x, Y = y) = f (x, y) = 1/36

Moreover, it is obvious that f (x, y) is greater than or equal to zero for all x and y, that

∑_{x=1}^{6} ∑_{y=1}^{6} f (x, y) = 1 and that P (2 ≤ X ≤ 4, 3 ≤ Y ≤ 5) = ∑_{x=2}^{4} ∑_{y=3}^{5} f (x, y)

and thus that f (x, y) satisfies the axioms for a probability density function.

Definition 42. Joint Distributions: The joint distribution of 2 random variables X and Y (or the joint distribution of the two dimensional random variable (X, Y) is specified by:

1. (discrete): Sets of possible values:

\begin{matrix}x_{1}, x_{2}..., x_{n} & for X \\ y_{1}, y_{2}..., y_{m} & for Y \end{matrix}

with probabilities associated with each point p (x_{i}, y_{j}) = P (X = x_{i}, Y = y_{j}), for i = 1, ..., n and j = 1, ..., m such that p (x_{i}, y_{j}) ≥ 0 and

∑_{i=1}^{n} ∑_{j=1}^{m} p (x_{i}, y_{j}) = 1

2. (a density function) f (x, y) such that f (x, y) ≥ 0 for all x and y and

∫_{−∞}^{∞} ∫_{−∞}^{∞} f (x, y) dxdy = 1

Definition 43. The distribution function in two-dimensions is defined as

F (x, y) = P (X ≤ x, Y ≤ y) = ∫_{−∞}^{x} ∫_{−∞}^{y} f (x, y) dxdy

Example 44. 2 coin tosses

X = \{ \begin{matrix}1 & (head) \\ 0 & (tail) \end{matrix} Y = \{ \begin{matrix}1 & (head) \\ 0 & (tail) \end{matrix}

Then P (1, 1) = P (1, 0) = P (0, 1) = P (0, 0) = 1/4.

Definition 45. Bivariate uniform distribution

f (x, y) = \{ \begin{matrix}1 & if x ∈ [0, 1] and y ∈ [0, 1] \\ 0 & otherwise \end{matrix}

Definition 46. Marginal Distribution (discrete): If X and Y have a joint distribution characterized by values:

\begin{matrix}x_{1}, x_{2}, ..., x_{n} \\ y_{1}, y_{2}, ..., y_{m} \end{matrix}

and probabilities p (x_{i}, y_{j}), then the marginal distribution of X is the distribution which assigns to the value x_{i} the probability p (x_{i}) = ∑_{j=1}^{m} p (x_{i}, y_{j}).

Example 47.

X = \{ \begin{matrix}1 & (head) \\ 0 & (tail) \end{matrix} Y = \{ \begin{matrix}1 & (head) \\ 0 & (tail) \end{matrix}

P (1) = P (1, 1)+p (1, 0) = 1/4+1/4 = 1/2; P (0) = P (0, 1)+p (0, 0) = 1/4+1/4 = 1/2

Definition 48. Marginal Distribution (continuous): If X and Y have a joint distribution with density function f (x, y), then the marginal distribution of X has the density function

f (x) = ∫_{−∞}^{∞} f (x, y) dy (note: F (x) and f (x, y) are not the same function)

Example 49. Let the joint distribution for the random variables (x,y) be given by:

f (x, y) = \{ \begin{matrix}1 & if x, y ∈ [0, 1] \\ 0 & otherwise \end{matrix}

Then the marginal density

f (x) = ∫_{−∞}^{∞} f (x, y) dy = ∫_{−0}^{1} f (x, y) dy = \{ \begin{matrix}1 & if x ∈ [0, 1] \\ 0 & otherwise \end{matrix}

Remark 10. There is no difference between the distribution of X and the marginal distribution of X derived from the joint distribution of X and Y.

Definition 50. Conditional Distribution: Let X and Y have a joint distribution. The conditional distribution of X given Y is characterized by

1. (discrete): P (X = x_{i} given Y = y_{j}) = P (x_{i}|y_{j}) = P (x_{i}, y_{j}) /P (y_{j}) for i = 1, 2, ⋯, n.

2. (continuous):

f_{x|y} (x|y) = \frac{f_{xy} (x, y)}{f_{y} (y)} for all x and y such that f_{y} (y) ≠ 0

Example 51. Population of a country is:Blond hair and blue eyes 20 million; Blond hair and brown eyes 10 million; Dark hair and blue eyes 5 million; Dark hair and brown eyes 65 million. Joint distribution: P (blond, blue) = 0.2; P (dark, blue) = 0.05; P (blond, brown) = 0.1; P (dark, brown) = 0.65. Marginal distributions: P (blond) = P (blond, blue) + P (blond, brown) = 0.2 + 0.1 = 0.3; P (dark) = 0.70; P (blue eyes) = 0.25; P (brown eyes) = 0.75. Conditional Distribution: P (blond|blue) = P (blond, blue) /P (blue) = 0.2/0.25 = 0.8.

Example 52.

f (x, y) = \{ \begin{matrix}1 & if x, y ∈ [0, 1] \\ 0 & otherwise \end{matrix}

and

f (y) = \{ \begin{matrix}1 & if y ∈ [0, 1] \\ 0 & otherwise \end{matrix}

Then

f_{x|y} (x|0.5) = \frac{f_{xy} (x, 0.5)}{f_{y} (0.5)} = \{ \begin{matrix}1 & if x ∈ [0, 1] \\ 0 & otherwise \end{matrix}

Remark 11. Multiplication Rule: P (x, y) = P (x) P (y|x) = P (y) P (x|y).

Example 53. Example: There are 10 balls in a jar numbered 1, 2, ⋯, 10. We pick one and do not put it back. Then pick another one. What is the probability that the first time is number 2 and the second time is number 6? P (2, 6) = P (2) P (6|2) = 1/10 × 1/9 = 1/90.

Example 54. (Bayes Rule)

P (x|y) = \frac{P (x) P (y|x)}{P (y)}

since

P (x|y) = \frac{P (x, y)}{P (y)} = \frac{P (x) P (y|x)}{P (y)}

Example 55. Suppose you have 10 jugs numbered 1, 2, ..., 10 and jug i has i white balls and 10− i black balls. Pick a jug at random and pick a ball from it. If the ball is white, what is the probability it came from jug 5?

P (x = 5|y = white) = \frac{P (x = 5) P (y = white|x = 5)}{P (y = white)} = \frac{1/10 × 1/2}{55/100} = 1/11

P (y = white) = ∑_{i=1}^{10} P (x = i, y = white) = ∑_{i=1}^{10} P (y = white|x = i) P (x = i) = \frac{55}{100}

Remark 12. E(X + Y) = E(X) + E(Y)

since

E(X + Y) = ∑_{i=1}^{n} ∑_{j=1}^{m} (x_{i} + y_{j}) P (x_{i} + y_{j}) = ∑_{i=1}^{n} ∑_{j=1}^{m} x_{i}P (x_{i} + y_{j}) + ∑_{i=1}^{n} ∑_{j=1}^{m} y_{j}P (x_{i} + y_{j})

= ∑_{i=1}^{n} x_{i}P (x_{i}) + ∑_{j=1}^{m} y_{j}P (y_{j}) = E(X) + E(Y)

Definition 56. The covariance of X and Y is

cov (X, Y) = E [(X − μ_{x}) (Y − μ_{y})] where μ_{x} = E(X) and μ_{y} = E(Y)

Remark 13. The covariance measures the independence or lack of it between two random variables. It can be negative, positive or zero. If positive, it indicates that when X −μ_{x} > 0 then Y −μ_{y} > 0 or when X −μ_{x} < 0 then Y −μ_{y} < 0. So a positive covariance means that big values of X go with big values of Y and small values of X go with small values of Y. Similarly a negative covariance means that big values of X go with small values of Y and small values of X go with big values of Y.

Remark 14. Another formula for the covariance is

cov (X, Y) = E (XY) − μ_{x}μ_{y}

since

cov (X, Y) = E [(X − μ_{x}) (Y − μ_{y})] = E [XY − μ_{x}Y − μ_{y}X + μ_{x}μ_{y}]
= E (XY) − E (Y) μ_{x} − E (X) μ_{y} + μ_{x}μ_{y} = E (XY) − μ_{x}μ_{y}

Example 57. Hair and eye color. Hair X = \{ \begin{matrix}1 & blond \\ 0 & dark \end{matrix} Eye Y = \{ \begin{matrix}1 & blue \\ 0 & brown \end{matrix}

P (1, 1) = .20 P (X = 1) = .30 μ_{x} = .30
P (1, 0) = .10 P (X = 0) = .70 μ_{y} = .25
P (0, 1) = .05 P (Y = 1) = .25
P (0, 0) = .65 P (Y = 0) = .75
E (XY) = 1 × 1P (1, 1) + 1 × 0P (1, 0) + 0 × 1P (0, 1) + 0 × 0P (0, 0) = .20
cov (X, Y) = E (XY) − μ_{x}μ_{y} = .20 − .30 × .25 = .125

Theorem 58. If X and Y are independent, then cov (X, Y) = 0.

Proof:

E (XY) = ∫_{−∞}^{∞} ∫_{−∞}^{∞} xyf (x, y) dxdy
= ∫_{−∞}^{∞} ∫_{−∞}^{∞} xyf (x) f (y) dxdy
= ∫_{−∞}^{∞} xf (x) dx ∫_{−∞}^{∞} yf (y) dy = E (X) E (Y)

Thus

cov (X, Y) = E (XY) − E (X) E (Y) = 0

Theorem 59.

var(X + Y) = var (X) + var (Y) + 2cov (X, Y)

Proof:

var(X + Y) = E [(X − μ_{x}) + (Y − μ_{y})]^{2}
= E (X − μ_{x})^{2} + E (Y − μ_{y})^{2} + 2E [(X − μ_{x}) (Y − μ_{y})]
= var (X) + var (Y) + 2cov (X, Y)

Remark 15. E (X + Y) = E (X) + E (Y) always Remark 16. var (X + Y) ≠ var (X) + var (Y) unless cov (X, Y) = 0, which is the case when X and Y are independent. Remark 17. E (XY) ≠ E (X) E (Y) unless cov (X, Y) = 0.

Definition 60. (Correlation Coefficient) The correlation coefficient of X and Y is ρ (X, Y) = \frac{cov(X,Y)}{SD(X)SD(Y)}

Remark 18. The sign of ρ (X, Y) is the same as the sign of cov (X, Y) since SD’s are always positive.

Lemma 61. If X has mean μ_{x} and variance σ_{x}^{2}, then X^{∗} = (X − μ_{x}) /σ_{x} has mean 0 and variance 1. X^{∗} is called the standardized variable corresponding to X.

Proof: E (X^{∗}) = 0 and var (X^{∗}) = var (X − μ_{x}) /σ_{x}^{2} = σ_{x}^{2}/σ_{x}^{2} = 1.

Lemma 62.

ρ (X, Y) = cov (X^{∗}, Y^{∗})

Proof:

cov (X^{∗}, Y^{∗}) = E (X^{∗}Y^{∗}) − E (X^{∗}) E (Y^{∗})

= E [(\frac{X − μ_{x}}{σ_{x}} ) (\frac{Y − μ_{y}}{σ_{y}} )] = \frac{1}{σ_{x}σ_{y}} cov (X, Y) = ρ (X, Y)

Theorem 63.

−1 ≤ ρ (X, Y) ≤ 1

Proof:

0 ≤ var (X^{∗} + Y^{∗}) = var (X^{∗}) + var (Y^{∗}) + 2cov (X^{∗}, Y^{∗})
= 1 + 1 + 2ρ (X, Y) = 2 [1 + ρ (X, Y)] ≥ 0

Thus

−1 ≤ ρ (X, Y)

Similarly

0 ≤ var (X^{∗} − Y^{∗}) = var (X^{∗}) + var (Y^{∗}) − 2cov (X^{∗}, Y^{∗})
= 1 + 1 − 2ρ (X, Y) = 2 [1 − ρ (X, Y)] ≥ 0

Thus

ρ (X, Y) ≤ 1

Remark 19. Use ρ (X, Y) or cov (X^{∗}, Y^{∗}) because the size of cov (X, Y) may be misleading. A large value of cov (X, Y) may be because both variances are large rather than because they are particularly well connected. ρ (X, Y) or cov (X^{∗}, Y^{∗}) standardizes the size of the variance so that you can get a measure of the correlation that is standardized.

Remark 20. Zero correlation does not imply independence:

−1 0 1
X
−1  1/6 1/3 1/6
0 Y 0 0 0
1  1/6 0 1/6

E (XY) = 0 and E (X) = 0, E (Y) = −1/3 ⟹ cov (X, Y) = ˙0. But P (X|Y = −1) ≠ P (X|Y = +1). Thus they are not independent.