Lectures On Mathematical Statistics-R. C. Sickles Revised September, 1999 1

Part I

Introduction to Probability and Statistics

Statistics forms a fundamental part of the technology of the scientific method. It provides tools for making decisions under conditions of uncertainty, for testing observable implications of scientific theories, and for quantifying models that are constructed to describe certain aspects of reality. These tools are of quite general applicability and are useful to the business executive and the government policy maker, as well as the research worker in any field of science.

For our purposes, statistics can be divided into two broad categories:

(1) Descriptive Statistics, which is concerned with summarizing, describing, and the reducing of data for analysis.

(2) Inferential Statistics, which is concerned with the process whereby data are used to make statements or decisions about the universe from which the data were obtained. The bulk of the course will deal with inferential statistics and thus when refer to statistics we will have inferential statistics in mind.

Within inferential statistics, we can make a further division into applied and theoretical statistics. The applied statistician is the one in the field, working either as a researcher themself or as a consultant to someone needing statistical advice. Faced with a practical problem, the applied statistician constructs a mathematical model that describes a situation as closely as possible, analyzes the model by mathematical methods, and then devises procedures to deal with the problem. In devising the procedures, they may develop a new methodology or use existing methods. However, they are always guided by the principles of the theory of statistics.

The theoretical statistician generally does not dirty her hands with actual data but rather is concerned with developing the general theory of statistics. She may be motivated by practical problems, but not necessarily. In general the broad advances in applied statistics in the past thirty years or so were made possible by remarkable developments in statistical theory which immediately proceeded them.

The theory of statistics is a branch of applied mathematics with its roots in the theory of probability which largely had been inspired by the study of games of chance (Pascal, 1623-1662; Bernoulli, 1654-1705).

Because statistics is essentially a mathematical discipline, this is the view we will adopt in these lectures. The purpose of the lectures is to acquaint you with a detailed and wide-ranging introduction to the laws of probability, their application to mathematical statistics, the basic results of the latter, and with statistical inference.

1. Random Events and the Basic Concepts of Probability On the basis of observation and experiment science attempts to formulate the natural laws of the phenomena that it studies. the simplest and most widely used such scheme is the following

Whenever a certain set of conditions K is realized, the event A occurs.

Ex. (a) If water under 760-mm atmosphereic pressure is heated over 212^{o}F (the set of conditions K), it is transformed into steam (the event A).

Ex. (b) The law of gravity, the law of the conservation of mass, and the law of supply and demand are other examples of such laws.

Types of events:

1. An event whose occurrence is inevitable whenever the set of conditions K is realized is said to be certain (sure).

2. An event that can never occur whenever the set of conditions K is realized is said to be impossible.

3. An event that may or may not occur whenever the set of conditions K is realized is said to be random.

In contrast to certain and impossible events, the information conveyed by the assertion that an event is random is of limited value. Because the event is neither certain nor impossible, we only know that the set of conditions K do not reflect all of the necessary and sufficient conditions for the event to occur.

A wide range of phenomena exist, however, for which, if the set of conditions K are realized repeatedly, the proportion of occurrences of event A seldom deviates significantly from some average value, and this number can therefore serve as a characteristic index of the mass phenomenon ( the repeated realization of the set of conditions K) with respect to event A. For such phenomena it is possible not only to state simply that event A is random but also to estimate in quantitative terms the chance of its occurrence. This estimate can be expressed by propositions of the form:

The probability that the event A will occur whenever the set of conditions K is realized is equal to P, where 0 ≤ P ≤ 1.

Laws of this kind are called probabilistic or stochastic and they play an important part in nearly every field of science.

Ex. Physics-no methods exist for predicting whether or not a given atom of radium will disintegrate within a given period of time, but on the basis of experimental data it is possible to determine the probability of this disintegration: An atom of radium disintegrates in a period of t years with the probability: p = 1 − exp {−0.000436t}. The set of conditions K in this example consist of the atom of radium not being subjected to any unusual external conditions, such as bombardment by fast particles, during the time under consideration. Event A is the disintegration of the radium atom during the given period of t years.

Ex. Economics-This one is somewhat more complicated than the previous example and is one of the best-established empirical regularities in economics. It has been observed, many times in many countries, that as the level of income per capita rises the proportion of income spent on food decreases. This is referred to as Engel’s Law (after Ernst Engel, a German economist and statistician who published the paper in 1857). This law does not say that everyone spends a smaller proportion of income on food as income rises or even that every time there is an increase in an economy’s per capita income the proportion of income spent on food falls. Rather, it simply states that there are forces in operation that cause consumers on the average to spend a decreasing proportion of a growing income on food. The set of conditions K is a increasing income, with other circumstances held reasonably constant (e.g., as absence of drought, abrupt changes in social structure, etc.), while the event A is that the proportion of income that is spent on food falls.

Ex. Rolling of the Die-When a homogeneous die has been rolled repeatedly it has been observed over a large number of occasions that each of the six faces appears about 1/6 of the time. Here the set of conditions K is the roll of the die, while the event A is the appearance of any particular face.

Note: The “P ” in the general statement of a stochastic law is a characteristic of the mass phenomenon in question, not a reflection of the state of mind of the observer. In other words, the probability is an objective quantity in a scientific sense since it is independent of the observer. This means that questions of the type:

1. What is the probability that every natural number greater than two can be expressed as the sum of two primes?

or

2. What is the probability that on June 25, 2077 Joe Smith will marry Judy Jones?

do not come under the purview of the general statement of stochastic laws since there is no objective basis for the probabilities involved. About the former, little is known at present, except that many mathematicians believe that it is very likely to be true; about the latter one can only assume that a definitive answer will be given on June 25, 2077.

However, there is a school of probability, whose members are willing, if their analyses requires it, to assign numbers to the probabilities in the two statements above. The subjective school of probability (Bayesian School) maintain that all probabilities must reflect judgments of the observer. We will for the most part consider probabilities to be objective characteristics of the mass phenomenon in question.

With the concepts of a random event and probability established, let us now inquire into how the probabilities are assigned or discovered. Historically, there have been two approaches, the classical (or a priori) method and the relative frequency (or statistical) method. These were joined in this century by the axiomatic approach. The axiomatic approach, because it is consistent with both the classical and relative frequency definitions and does not suffer their defects, has essentially carried the day and will be the approach that we will follow. Before presenting these approaches, we need to appropriate some concepts from the algebra of sets.

1.1. Terms and Definitions from the Algebra of Sets.

Definition 1. Sets A and B are mutually exclusive if A ∩ B = ∅.

Thus A ∩ A′ = ∅.

Definition 2. If we have n sets A_{i}, i = 1, ..., n that are pairwise mutually exclusive, i.e. A_{i} ∩ A_{j} = ∅, ∀ i ≠ j and ∪_{i=1}^{n}A_{i} = S, then the sets A_{i} form an exhaustive partition of S.

Suppose that we now have n sets E_{i}, i = 1, ..., n that form an exhaustive partition of S. We call the E_{i}’s elementary events and S the space containing the events (sample space). For example, assume that n = 3. Then we can construct the events as ∅, E_{1}, E_{2}, E_{3}, E_{1} ∪ E_{2}, E_{1} ∪ E_{3}, E_{2} ∪ E_{3} and E_{1} ∪ E_{2} ∪ E_{3}. Each of these events can be decomposed into elementary sets and there are 8 of these events in all. ∅ is the null set while E_{1} ∪ E_{2} ∪ E_{3} is equal to S. These 8 events, or subsets, exhaust the nonredundant subsets and illustrate the fact: if a set S can be partitioned into n sets that are mutually exclusive, then S contains 2^{n} unique subsets.

Ex. 1- Flipping a coin interpreted in the algebra of sets:

There are two possible outcomes. Therefore S contains 2 elementary events, E_{1} = H and E_{2} = T. The 2^{2} subsets are ∅, H, T, and H ∪ T.

Ex. 2- Rolling a die interpreted in the algebra of sets:

S contains six elementary events: E_{1}: one dot-face shows; E_{2}: two-dot-face shows; E_{3}: three-dot-face shows; E_{4}: four-dot-face shows; E_{5}: five-dot-face shows; E_{6}: sixdot-face shows. There are 2^{6} events that can be constructed in all including the impossible event.

Recalling our definition of a random event, it is clear that the events in these two examples are random. The conditions K are, respectively, the flipping of a coin and the rolling of a die. The random events are showing of H or T and the showing of one of the six faces of the die. Moreover, it is well established that if K is realized repeatedly then the proportion of times that an event occurs will seldom deviate from some average value, i.e., a probability (P) can be associated with such event.

We will next study how these probabilities are determined.

2. Classical Notion of Probability

The classical definition of probability reduces to the concept of equal likelihood, which is taken as a primitive notion and therefore not subject to formal definition. For example, if a die is a perfect cube and made of a homogeneous substance, then there is no objective basis for assuming that any one face is more probable than any other. Similarly, if a coin is balanced, each face is considered to be equally probable.

In the general case, suppose that we have n pairwise mutually exclusive elementary events, E_{i}, i = 1, ..., n, that are equally likely. Let us now form the family F of events consisting of the impossible events ∅, the certain event S and all events which can be decomposed into a union of elementary events. F contains 2^{n} such events (such a family of events is called a field).

The classical definition of probability is given in terms of the events of family F and may be formulated as:

Definition 3. If an event A is decomposable into the sum of m elementary events belonging to a complete group of n pairwise mutually exclusive and equally likely events, then the probability P (A) of the event A is equal to

P (A) = \frac{m}{n}

Ex. Rolling a die- if A = E_{1} ∪ E_{2} ∪ E_{3}, then P (A) = 3/6 = 1/2.

In the study of probability, the terminology is fairly standard. First imagine that in order to clear up the question as to whether or not a particular event will occur it is necessary to make a trial (i.e., to realize the set of conditions K) which would yield the answer to our question (in the past example a die was to be thrown). The complete group of pairwise mutually exclusive and equally likely events which may occur when this experiment is performed is called the complete group of possible outcomes of the trial. The possible outcomes into which the event A can be decomposed are called the outcomes (or cases) favorable to A. Using this terminology, we can say that the probability P (A) of the event A is equal to the number of possible outcomes favorable to A divided by the total number of possible outcomes of the trial.

According to the definition given, every event belonging to the family of events F, as just described, has a well-defined probability

P (A) = \frac{m}{n}

assigned to it, where m is the number of mutually exclusive and equally likely events E_{i}, of F into the union of which the events A can be decomposed. Thus, the probability P (A) can be regarded as the real-value function of the event A defined over the field of events F.

This function possess the following properties:

1. For every event A of the field F, P (A) ≥ 0.

2. For the certain event S, P (S) = 1.

3. If the event A can be decomposed into the mutually exclusive events B and C and all three of the events A, B, and C belong to the field F, then P (A) = P (B) + P (C). This property is called the theorem on the addition of probabilities.

Remark 1. Property 1 is obvious, since the ratio m/n can never be negative, and property 2 is equally obvious, since the n possible outcomes are all favorable to the certain event S and hence,

P (S) = n/n = 1.

As for property 3, suppose that k is the number of outcomes favorable to the event B and that r is the number of outcomes favorable to event C. Because the events B and C are, by definition, mutually exclusive, the outcomes that are favorable to the event A = B ∪ C are equal k + r. Hence,

P (A) = (k + r)/n = k/n + r/n = P (B) + P (C)

A few other properties of probabilities are:

4. P (A′) = 1 − P (A)

Proof: A ∪ A′ = S. From Property 2, P (A ∪ A′) = P (S) = 1. Since A and

A′ are mutually exclusive, from Property 3, we have P (A ∪ A′) = P (A) + P (A′). Thus P (A′) = 1 − P (A).

5. P (∅) = 0

Proof: S and ∅ are mutually exclusive and thus P (S ∪ ∅) = P (S) + P (∅) = P (S). Thus P (∅) = 0.

6. If event A implies event B (i.e. A is a subset of B), then P (A) ≤ P (B).

Proof: Since A ⊂ B we have B = A ∪ A′B. Thus P (B) = P (A) + P (A′B) ≥ P (A).

7. The probability of any event lies between 0 and 1.

8. For any events A, B and C included in F, P (A ∪ B) = P (A)+P (B)−P (AB). And P (A ∪ B ∪ C) = P (A)+P (B)+P (C)−P (AB)−P (AC)−P (BC)+P (ABC)

If for no reason other than its elegant simplicity, the classical definition of probability has a great deal of appeal and, so long as the study of probability is the study of the outcomes of games of chance in finite sample spaces, there was little apparent need for any other definition. Nevertheless, there are some rather troublesome defects in the classical definition. It is obvious, e.g. that the classical definition has to be modified somehow if the number of possible random events is infinite, For instance we might seek the probability that a positive integer drawn at random is even. The intuitive answer to this is that the probability is 1/2, and if pressed to justify this answer on the basis of the classical definition, we might reason as follows:

Conjecture 4. Of the first 10 integers, 5 are even, hence the probability is 1/2; of the first 100, 50 are even, again the probability is 1/2, and so on. in general, of the first 2N positive integers, N are even, so that if we form the ratio N/2N and let N approach infinity so as to encompass all positive integers, the ratio, and therefore the probability of an even integer, remains at 1/2.

Although the argument is plausible, it is not easy to make it stand up. For one thing, it depends on the natural ordering of the positive integers and a different ordering would produce a different result. We could, e.g., take the following ordering: 1, 3, 2, 5, 7, 4, 9, 11, 6, ..., in which case the probability would appear to be 1/3. We could, just by choosing the right ordering, obtain any probability we desired. Since the integers can be ordered we can make the ratiol oscillate and it will never approach a definite value as N increases.

However, there is a even more fundamental difficulty with the classical definition of probability than that encountered with an infinity of possible outcomes. Consider the flipping of a coin or the rolling of a die that are known to be make out of inhomogeneous materials. The two possible outcomes of flipping a coin and the six possible outcomes of rolling the die are no longer equally likely. How do we calculate the probabilities? The classical definition is completely helpless in these situations.

Still another difficulty with the classical definition of probability is the calculation of subjective probabilities when we try to answer questions of the following type:

What is the probability that a man living in Sweden will live to age 60?

What is the probability that a light bulb will burn 100 hours or less?

What is the probability that two cars going through the same intersection will collide?

What is the probability that a woman’s income known to be above $100,000 per year in the past is above $150,000 this year?

All of there are legitimate questions that we would like to bring into the realm of probability. They are impossible to answer using the classical definition. Questions involving ”symmetry” and ”equally likely” can not be answered as in games of chances. The classical definition must be altered or extended if these and similar questions are to be brought into the framework of the theory. The relative frequency (or statistical) definition of probability partially allows for this to be done.

2.1. Relative Frequency Definition of Probability. Lengthy observations of the occurrence or nonoccurrence of an event A in a large number of repeated trials under the same set of conditions K show that for a wide class of phenomena the number of occurrences or nonoccurrences is subject to a stable law. If we denote by m the number of times that A occurs in a sequence of n independent trials, then it turns out that for sufficiently large n the ratio m/n in most of such series of observations assumes an almost constant value, with large deviations being less frequent the larger the number of trials.

This kind of stability of the relative frequency (i.e. of the ratio m/n) was first observed in demographic phenomena. In ancient times it was already noticed that the ratio of male to the total number of births in entire countries and in large cities varied little from one year to the next,. In the seventeenth and eighteenth centuries, a number of fundamental works devoted to the study of population statistics appeared. Laplace, for example, in his extensive analyses of continental birth statistics discovered that during one decade the ratio of male to total births for London, St. Petersburg, Berlin, and all of France all fluctuated around one and the same number, which was approximately 22/43. Apart from the stability of the ratio of male to female births, the researchers of that period also discovered that demographic laws of another character existed: the percentage of deaths at a particular age in certain groups of the population (of a given economic and social background), the distribution of people (of a definite sex, age, and nationality) according to height, weight, breadth of chest, length of footstep, and so on.

Moreover, it turns out that in all those cases in which the classical definition of probability is applicable, the fluctuation of the relative frequency is in the neighborhood of the probability P of the event. There is a vast amount of experimental evidence in verification of this fact resulting from experiments involving coin tossing, dice rolling, and so on. The results from three such coins’ tossing experiments are recorded below:

Name of experimenter # Tosses # Trials Rel. Freq.
Buffon  4, 040  2, 048  0.5080
Karl Pearson  12, 000 6, 019  0.5016
Karl Pearson  24, 000 12, 012 0.5005

The fact that in a number of instances the relative frequency of random events in a large number of trials is almost a constant compels us to presume the existence of certain laws, independent of the experimenter, that govern the course of the phenomena. The fact that the relative frequency of an event to which the classical definition is applicable is, as a rule, close to its probability whenever the number of experiments is large is strong evidence that in the general case as well there exists some constant about which the relative frequency fluctuates. Because this constant is an objective numerical characteristic of the phenomena, it is natural to call it the probability of the random event A under investigation.

Definition 5. Definition 6. A random event A has a probability P (A) if it satisfies the following conditions:

1. It is possible, at least in principle, to make an unlimited number of independent trails under the same set of conditions K, in which the event A may or may not occur.

2. As a result of a sufficiently large number of trials, the relative frequency in nearly every one of a large group of sequences of trials is observed to deviate only negligibly from a certain (but generally unknown) constant.

We may take as a approximation to this constant the relative frequency of the event in a large number of trials, or else some value reasonably close to the relative frequency. Hence, the probability should have the following properties:

1. The probability of the certain event is a one.

2. The probability of the impossible event is zero.

3. If a random event C is the sum of a finite number of mutually exclusive events A_{i}, i = 1, ..., n, each with a probability, then the probability of C exists and is equal to the sum of the probabilities of the events A_{i}:

P (C) = P (A_{1}) + P (A_{2}) + ⋯ + P (A_{n})

An important attribute of the relative frequency definition of probability is that it retains a character that is independent of the experimenter. The fact that the probabilities can be inferred only after observation in no way reduces the value of deductions nor lessens the objectivity of the probabilities.

Nevertheless, despite the fact that the relative frequency definition overcomes some of the problems of the classical definition, it has some drawbacks of its own. In the first place, it is more in the way of a description than of a formal mathematical character. Moreover, it does not reveal to any extent, although this is less serious, the nature of the phenomena being investigated and for which the relative frequency is stable. Finally, the relative frequency definition is totally inadequate in situations where probabilities are viewed as representing reasonable degrees of belief.

Thus, from a formal point of view the mathematical foundation of the theory of probability around the turn of this century was unsatisfactory. This had not kept natural scientists with their rather naive concepts of probability from many important successes, but clearly what was needed was a method for systematically studying the fundamental concepts of probability and for clarifying the conditions under which the results of the theory could be applied. The theory needed to be based on a set of premises which generalized the centuries of experience with probability and form which the development of the theory could proceed by logical deduction without recourse to intuitive notions of ”common sense.” It is to this that we now turn.

3. Axiomatic Formulation of the Theory of Probability An axiomatic approach to probability theory was first expressed in 1917 in a paper by the Russian mathematician, S. N. Bernstein. However, the approach that we shall follow here is the one of another Russian mathematician, Kolmogorov, and closely relates probability theory to the theory of sets and the modern measure-theoretic aspects of the theory of functions of a real variable. The strength of Kolmogorov’s formulation is that it embodies all of the fundamental properties of probability observed in the classical and relative frequency definitions, yet at the same time has met all of the demands placed on the theory by modern science, not only the natural sciences, but the social sciences as well.

Kolmogorov’s Axioms

We begin with a set Ω consisting of elementary events (what the elements of this set are is immaterial to the logical development of the theory) and consider a certain family F of subsets of this set. The elements of the family F shall be called random events. The following three conditions are imposed on the structure of the family F:

1. F contains the set Ω as one of its elements.

2. If the subsets A and B are included in F, then so also are A ∪ B, A′ and B′. Because Ω is included in F, the second requirement implies that F includes Ω′; that is, the empty set is included in F. Moreover, it is easy to see that the second requirement implies that the union and complements of a finite number of random events of F are also included in F. In other words, F is closed under the operations of union and complementation. Such a set is called a field of events.

3. If the infinite sequence of subsets A_{1}, A_{2}, ⋯, A_{n} of Ω are included in F, then the union A_{1} ∪A_{2} ∪···∪A_{n} and intersections A_{1}A_{2} ···A_{n} of these subsets are included in F.

Remark 2. This condition means that the field also contains unions and intersections of all countable sequences of sets. A field that satisfies such a condition is called a Borel field (σ-field or σ-algebra)

Ex. Tossing a coin once. The elementary events are Ω = {H, T }; random events are F = {Ω, ∅, H, T }.

Ex. Rolling a die once. The elementary events are Ω = {E_{1}, E_{2}, E_{3}, E_{4}, E_{5}, E_{6}}; random events are F = {2^{6} = 64 = # of all the subsets of Ω}.

Remark 3. If A ∩ B = ∅, then A and B are mutually exclusive. Random events Ω and ∅ are called the sure and impossible events.

We now formulate the Axioms that define probability

Axiom 1. 1: With each random event A in the field of events, there is associated a nonnegative number P (A) called its probability.

Axiom 2. 2 P (Ω) = 1.

Axiom 3. 3 (addition axiom) If the events A_{1}, A_{2}, ⋯, A_{n} are mutually exclusive, then P (A_{1} ∪ A_{2} ∪ ⋯ ∪ A_{n}) = P (A_{1}) +P (A_{2}) + ⋯ + P (A_{n}).

Remark 4. It wasn’t necessary to postulate Axioms 1, 2 and 3 since they could be derived as theorems with the classical definition. The first axiom is contained in classical definition.

Axiom 4. 4. (Extended Axiom of addition) If the event A is equal to the union of a infinite sequence A_{1}, A_{2}, ⋯, and the events in this sequence are pairwise exclusive then

P (A) = P (A_{1}) + P (A_{2}) + ⋯

Proposition 7. We need Axiom 4 to handle cases in which the number of elements is infinite.

Let’s deduce some important consequences of these 4 Axioms. The proofs of some of them are the same as above.

1. P (∅) = 0

2. P (A′) = 1 − P (A)

3. 0 ≤ P (A) ≤ 1

4. If A ⊂ B, then P (A) ≤ P (B)

5. Let A and B be arbitrary events, then P (A ∪ B) = P (A) + P (B) − P (AB) Since P (AB) ≥ 0, we have

6. P (A ∪ B) ≤ P (A) + P (B)

By induction we have

7. P (A_{1} ∪ A_{2} ∪ ⋯ ∪ A_{n}) ≤ P (A_{1}) +P (A_{2}) + ⋯ + P (A_{n})

These system of axioms is consistent since we can find real objects that satisfy the axioms.

Ex. Let Ω be finite Ω = {a_{1}, a_{2}, ⋯, a_{n}} and F = {all the subsets of Ω denoted by B_{1}, B_{2}, ⋯, B_{k}}. We can satisfy the axioms by setting P (a_{i}) = p_{i} ≥ 0 and ∑_{i=1}^{n} p_{i} = 1. Then

P (B_{i}) = ∑_{j=i_{s}} p_{j}

where i_{s} is an index set designation that the elements a_{j} that are included in B_{i}.

Remark 5. Note that the system of axioms is incomplete, since for a given set Ω we can select probabilities in the set F in different ways.

Ex. Rolling a die.

We can choose P (E_{1}) = P (E_{2}) = P (E_{3}) = P (E_{4}) = P (E_{5}) = P (E_{6}) = 1/6 or P (E_{1}) = P (E_{2}) = 1/6, P (E_{3}) = P (E_{4}) = 1/4 and P (E_{5}) = P (E_{6}) = 1/12.

The incompleteness of the set of axioms does not mean that the choice of axioms is an unfortunate one or that not enough thought has gone into their development, but rather is in the nature of things. Often there are situations where it is necessary to study the same random events, but with different probabilities. For example, the first system of equations above would be the probabilities associated with a die the is true. The second system of equations, on the other hand, might be the probabilities associated with a ”loaded’ die. Therefore, the incompleteness of the set of axioms is necessary for the practical usefulness of the theory of probability.

3.1. Conditional Probability. Thus far it has been taken for granted that in speaking of the probability of an event A it has been with reference to the entire sample space Ω. In many instances, however, we will encounter problems of the following type: What is the probability of event A given that event B has already occurred?

Ex. 1. Suppose that a pair of dice are thrown. What is the probability that one of the faces is a 3 given that sum of the two faces is 9?

Ex. 2. Suppose that a card is drawn from a well-shuffled bridge deck. What is the probability that the card is an ace given that it is a space?

Ex. 3. Suppose that 10 people are drawn at random from a city in Canada. What is the probability that they are of Italian descent given that the city is Toronto?

Ex. 4. Brand preference/previous purchase of new products.

In each of these examples it is evident that the probability asked for is with reference to a restricted sample space. In the first example, instead of the original sample space of 36 elementary events, the relevant sample space consists only of those 4 events which give a sum of 9. Similarly, the relevant sample space in the second example consists of the 13 spades, while in the third we need only consider the distribution of nationalities within Toronto. In each case, the restricted sample space is equal to the intersection of event B with the original sample space Ω, that is that part of Ω that is also B.

Probabilities with reference to a restricted sample space are called conditional probabilities. It s evident that the probability (relative frequency) of A given that B has already occurred, written as P (A|B) is

P (A|B) = P (AB) /P (B)

In other words, the probability of A given B is equal to the ratio of the probability of the simultaneous (or joint) occurrence of A and B to the probability of B. It is assumed, of course, that P (B) is not 0. It should be noted that we can interpret all probability as conditional probability, since, with reference to the unrestricted sample space Ω, we have

P (A|Ω) = P (AΩ) /P (Ω) = P (A)

Ex. Consider the experiment of throwing a pair of dice, for which the sample space is below. Let the sum of the two faces be denoted by R.

Die 2 (F_{2})
(1, 1) (2, 1) (3, 1) (4, 1) (5, 1) (5, 1)
(1, 2)
Die 1 (F_{1}) (1, 3)
(1, 4)
(1, 5)
(1, 6)  (6, 6)

1. What is P (F_{1} = 3|R = 9)? P (R = 9) = 1/36 and P ((F_{1} = 3) ∩ (R = 9)) = 1/36. Thus P (F_{1} = 3|R = 9) = (1/36) /(4/36) = 1/4.

2. What is P (F_{1} ≤ 4|R = 7)? Since P (R = 7) = 6/36 and P {(F_{1} ≤ 4) ∩ (R = 7)} = 4/36. Thus P (F_{1} ≤ 4|R = 7) = (4/36) / (6/36) = 2/3.

Independent Events.

Definition: A is independent of B if P (A/B) = P (A).

Along with the definition of conditional probability, we have P (AB) = P (A)P (B) and P (B/A) = P (B).

Note: A and B′ are independent if A and B are independent.

Proof: P (AB) = P (A−AB) = P (A)−P (AB) = P (A)−P (A)P (B) = P (A) (1 − P (B)) = P (A)P (B′).

Example of independent events: If a card drawn at random is a Spade, what is the probability that it is an Ace?

P (Spade) = 13/52 = 1/4; P (Spade and Ace) = 1/52; P (Ace) = 4/52 = 1/13. Thus P (Spade and Ace) = P (Spade) · P (Ace). So events are independent.

In the rolling of a pair of true dice, the nature of the experiment suggests that the face showing on the second die should not be influenced by the face on the first, so that, e.g., the probability of rolling two 6’s is equivalent to the products of the probabilities of a 6 on each die. Generally, however, it is impossible to tell whether two events are independent simply by looking at them or considering their nature. Thus, to check independence we usually have to calculate the probabilities, as in the immediately preceding example.

It is important not to confuse independence with mutual exclusivity. Two events that are mutually exclusive are not independent since the sets must overlap in order to be independent, as the example of the event A and its complement A′ will readily show. A possible source of confusion is the identifying of mutually exclusive events with the common expression “having nothing to do with each other.” This is a useful description of independence when applied to everyday events, but is mistaken when applied to sets for it suggests nonoverlapping. For two sets to be independent it is necessary that they have an overlap; that is, A intersect B must not be equal to the empty set.

It is natural to wish to extend the notion of independence to three or more events by requiring that the probability of their intersection be equal to the product of their probabilities, namely,

P (A_{1} ∩ A_{2} ∩ ⋯ ∩ A_{n}) = P (A_{1}) P (A_{2}) ⋯ P (A_{n})

Such a requirement is not sufficient however. We say that n events are completely independent if and only if every combination of these events, taken any number at a time, is independent, i.e., A_{1}, A_{2}, A_{3} are independent if and only if the followings hold

P (A_{1} ∩ A_{2} ∩ A_{3}) = P (A_{1}) P (A_{2}) P (A_{3})
P (A_{1} ∩ A_{2}) = P (A_{1}) P (A_{2})
P (A_{1} ∩ A_{3}) = P (A_{1}) P (A_{3})
P (A_{2} ∩ A_{3}) = P (A_{2}) P (A_{3})

Remark 6. If these equations are satisfied then so are the ones obtained by replacing any of the events by their complements, e.g.

P (A_{1} ∩ A2^{′} ∩ A3^{′}) = P (A_{1}) P (A2^{′}) P (A3^{′})

4. Bayes Theorem

Suppose that we pose the following situation: A ball is drawn from one of two urns depending on the outcome of a roll of true die. If the die shows a 1 or 2, the ball is drawn from Urn I which contains 7 red balls and 3 white balls. If the die shows a 3, 4, 5 or 6, the ball is drawn from Urn II which contains 4 red balls and 6 white balls. Given that a red ball is drawn, what is the probability that it came from Urn I? Urn II?

To compute these probabilities, we proceed as follows: First, denote the urns by I and II and balls by R and W. We therefore have

\begin{matrix}P (I) = 1/3 & P (II) = 2/3 & P (R|I) = 7/10 \\ P (W |I) = 3/10 & P (R|II) = 4/10 & P (W |II) = 6/10 \end{matrix}

and we want P (I|R) and P (II|R). By definition of conditional probability

P (I|R) = P (I intersect R) /P (R)

But since I and II are mutually exclusive and exhaustive events,

P (R) = P (R ∩ I) + P (R ∩ II)
P (I ∩ R) = P (R|I) P (I)
P (II ∩ R) = P (R|II) P (II)

Therefore

P (I|R) = \frac{P (R|I) P (I)}{P (R|I) P (I) + P (R|II) P (II)}

= \frac{(7/10)(1/3)}{(7/10)(1/3) + (4/10)(2/3)} = 7/15

The probabilities obtained for the urns after the result of the drawing are often referred to as posterior (or after the fact) probabilities in contrast to the unconditional probabilities which are called prior (or before the fact) probabilities.

In this example, the fact that the ball drawn was red substantially increases the probability that it came from Urn I.

\begin{matrix}Urn & P rior prob. & P osterior prob. given R & P osterior prob. given W \\ I & 1/3 & 7/15 & 1/5 \\ II & 2/3 & 8/15 & 4/5 \end{matrix}

The generalization of these formula to n events is called Bayes theorem on inverse probabilities and is stated as follows:

Theorem 8. (Bayes Theorem) If the events A_{1}, A_{2}, ⋯, A_{k} constitute a partition of the sample space and P (A_{i}) ≠ 0 for i = 1, 2, ..., k, then for any event B in the sample space

P (B) = ∑_{i=1}^{k} P (A_{i}) P (B|A_{i})

and

P (A_{i}|B) = \frac{P (A_{i}) P (B|A_{i})}{∑_{i=1}^{k} P (A_{i}) P (B|A_{i})}

Proof:

P (B) = P (B ∩ (∪_{i}^{k}A_{i})) = P (∪_{i}^{k}BA_{i})

= ∑_{i=1}^{k} P (BA_{i}) = ∑_{i=1}^{k} P (A_{i}) P (B|A_{i})

P (A_{i}|B) = \frac{P (A_{i}B)}{P (B)} = (P (A_{i}) P (B|A_{i})) / (∑_{i=1}^{k} P (A_{i}) P (B|A_{i}))

Ex. Population of men and women, each of whom expresses a preference for brand for brand A or B of a firm’s product

W  M
A  B  A  B
50%  50% 30%  70%

Assume that 45% of population are women, what is P (W |A)? P (W |B)?

P (W |A) = \frac{P (A|W) P (W)}{P (A|W) P (W) + P (A|M) P (M)}

= \frac{0.5 × 0.45}{0.5 × 0.45 + 0.3 × 0.55} = 0.577

P (W |B) = \frac{P (B|W) P (W)}{P (B|W) P (W) + P (B|M) P (M)}

= \frac{0.7 × 0.45}{0.5 × 0.45 + 0.7 × 0.55} = 0.631

Ex. Suppose that the Toys for Tots company is contemplating a new advertising campaign to increase its share of the market. The marketing people in the company assume that the increase in market share can be 0%, 5% or 10%. These are the three states of nature. However, before the final stages of the new campaign are undertaken (that is, before it is too late to back out). Toys for Tots will be able to observe whether its main competitor is about to undertake a similar campaign. If the competitor does undertake its campaign. Toys for Tots assesses the probabilities of its three possible increases to 0.5, 0.3 and 0.2. If the competitor does not undertake its campaign, Toys for Tots believes that the probabilities of these three possible increases are 0.1, 0.4, and 0.5.

In this example a likelihood would be of the form

P (competitor′s plan|increase in T oys for T ots′s market share)

and Bayes’ rule would be used to convert this to a posterior probability of the form

P (increase in T oys for T ots′s market share|competitor′s plan)

However, the likelihoods are not given in the problem statement. Instead, the posterior probabilities are given directly. Therefore, when Toys for Tots is making its final stage decision, after observing its competitor’s plan, it does not need to use Bayes’ rule; the probabilities it needs are exactly tosses that are given.

The use of Bayes theorem form one of the more controversial areas in all of statistics. As it stands it is true because it is a mathematical theorem. However, trouble arises when it comes to determining the set of events A_{i} which form an exhaustive partition of the sample space and assigning them probabilities. In most real life instances where we would like to apply Bayes theorem it is impossible to establish for certain whether A_{i} in fact forms an exhaustive partition, e.g., we may have a situation where it is possible to associate via probabilities certain medical symptoms with certain diseases. We would like to be able to turn this around and calculate via Bayes theorem the probability of a particular disease given some particular symptoms. However, to do so would require an exhaustive list of the causes of the symptom and their appropriate probabilities. Such knowledge is usually lacking.

Problems such as the one just noted, however, have not stood in the way of Bayes theorem achieving widespread use. Indeed, there is one entire school of statistics, the so-called Bayesian school, for which Bayes theorem provides the point of departure. Almost by necessity, Bayesian statistics requires the use of subjective probability. We delay our treatment of this until a latter set of lectures

We have now discussed the concept of probability from three points of view and have proved the key theorem, and it is time to begin devising ways of systematically assigning probabilities. In most cases, this is done by means of a probability function, and tour study of probability will reduce to the study of particular probability functions and their properties.