Lectures on Mathematical Statistics-R. C. Sickles

September 29, 1999

1 Asymptotic Theory

1.1 The Convergence of Random Variables

Let Ω be the set of possible outcome of an experiment. Let w be a particular outcome (w ∈ Ω). Let X (w) be a random variable: a real-valued function with domain Ω that assigns a numerical value to each outcome.

Definition 1 A sequence of r.v.’s {X_{n}, n ∈ ℵ} with distribution functions {F_{n} (x) , n ∈ ℵ} is said to converge in distribution to X, denoted by X_{n} →^{d} X, if

lim_{n→∞} F_{n} (x) = F (x)

at every continuity point x of F (x).

Remark 1 X_{n} (w) and X (w) may be scalars or column vectors..

Definition 2 A sequence of r.v.’s {X_{n}, n ∈ ℵ} is said to converge in probability to X, denoted by X_{n} →^{p} X, if for any small positive ε

lim_{n→∞} Pr (w : |X_{n} (w) − X (w)| < ε) = 1

Remark 2 X_{n} (w) and X (w) may be scalars or column vectors. In the latter case, |X_{n} (w) − X (w)| should be interpreted as the Euclidean norm. Also X can be a r.v. or a constant.

Definition 3 A sequence of r.v.’s {X_{n}, n ∈ ℵ} is said to converge almost surely (a.s.) to X, denoted by X_{n} →^{a.s.} X, if for any small positive ε

lim_{n→∞} Pr (w : max_{m≥n} |X_{m} (w) − X (w)| < ε) = 1

An equivalent way of defining almost sure convergence is by

Pr (w : lim_{n→∞} X_{n} (w) = X (w)) = 1

Note: Convergence in probability is the mode of convergence associated with the (weak) law of large numbers (LLN). The almost sure convergence is the mode of convergence associated with the strong law of large numbers (SLLN). From the definition of both sorts of convergence, we know almost sure convergence implies weak convergence. Another mode of convergence considered in relation to the limit theorem is convergence in rth mean.

Definition 4 Let {X_{n}, n ∈ ℵ} be a sequence of r.v.’s such that E (|X_{n}|^{r}) < ∞ for all n ∈ ℵ and E (|X|^{r}) < ∞ for r > 0, then the sequence is said to converge X in rth mean, denoted by X_{n} →^{r} X,

lim_{n→∞} E(|X_{n} (w) − X (w)|^{r}) = 0

Remark 3 The two special cases, r = 1 (the convergence in mean) and r = 2 (the convergence in mean square), are of particular interest.

The relations of these four sorts of convergences are demonstrated by the following theorem:

Theorem 5

→^{a.s.} ; ↘ ; →^{p} ⇒ →^{d} ; ↗ ; →^{r}

Proof: That almost sure convergence implies convergence in probability is obvious.

That convergence in rth mean implies convergence in probability is demonstrated as follows

Pr (|X_{n} − X| > ε) = Pr (|X_{n} − X|^{r} /ε^{r} > 1)
≤ E (|X_{n} − X|^{r} /ε^{r})
= \frac{1}{ε^{r}} E(|X_{n} − X|^{r}) → 0

That convergence in probability implies convergence in distribution is proved as follows. For any δ > 0,

lim_{n→∞} Pr(X_{n} ≤ t)

= lim_{n→∞} Pr(X_{n} ≤ t; X ≤ t + δ, X > t + δ)
= lim_{n→∞} Pr (X_{n} ≤ t, X ≤ t + δ) + \overbrace{lim_{n→∞} Pr (X_{n} ≤ t, X > t + δ)}^{0}
= lim_{n→∞} Pr (X_{n} ≤ t, X ≤ t + δ) ≤ Pr (X ≤ t + δ)

Using the same technique, we can obtain

lim_{n→∞} Pr (X_{n} ≤ t) ≥ Pr (X ≤ t − δ)

Thus, we have

Pr (X ≤ t − δ) < lim_{n→∞} Pr (X_{n} ≤ t) ≤ Pr (X ≤ t + δ)

which is

F (t − δ) < lim_{n→∞} F_{n} (t) ≤ F (t + δ)

Remember that the above inequality holds for all δ > 0, let δ ↓ 0, if F (t) is continuous at t, we have

lim_{n→∞} F_{n} (t) = F (t)

Example 6 Convergence in distribution does not imply convergence in probability. Assume X ∼ N (0, 1). Define a random sequence as X_{n} = −X. Then

X_{n} →^{d} X but not X_{n} →^{p} X.

Example 7 Convergence in probability does not imply convergence in rth moment. Without loss of generality, we assume r = 1. Define X_{n} and X as follows: X = 0 and

X_{n} = \{ \begin{matrix}n^{2} & with p = 1/n \\ 0 & with p = 1 − 2/n \\ −n^{2} & with p = 1/n \end{matrix}

We can see that X_{n} →^{p} X, but not X_{n} →^{convergence in mean} X.

Example 8 The example that convergence in probability does not imply convergence almost surely is very complicated and omitted here.

Theorem 9 Let {X_{n}, n ∈ ℵ} be a sequence of r.v.’s and g (·) : ℜ → ℜ a continuous function, then:

X_{n} →^{a.s.} X ⇒ g (X_{n}) →^{a.s.} g (X)

X_{n} →^{p} X ⇒ g (X_{n}) →^{p} g (X)

X_{n} →^{d} X ⇒ g (X_{n}) →^{d} g (X)

Proof: First claim is trivial. Second claim can be proved as follows

lim_{n→∞} Pr (|g (X_{n}) − g (X)| > ε)
= lim_{n→∞} Pr (|g (X_{n}) − g (X)| > ε; |X_{n} − X| ≤ δ, |X_{n} − X| > δ)
= lim_{n→∞} Pr ((|g (X_{n}) − g (X)| > ε; ) |X_{n} − X| ≤ δ ∪ |g (X_{n}) − g (X)| > ε; |X_{n} − X| > δ)
= \overbrace{lim_{n→∞} Pr (|g (X_{n}) − g (X)| > ε; |X_{n} − X| ≤ δ)}^{0}
= lim_{n→∞} Pr (|g (X_{n}) − g (X)| > ε; |X_{n} − X| > δ)
≤ lim_{n→∞} Pr (|X_{n} − X| > δ) = 0

The 4th line, which equals zero, of this proof uses the fact that g (·) is uniformly continuous. If it is not uniformly continuous, we have to divide the real line into a closed interval, on which g (·) is uniformly continuous, and the rest on both sides of this closed interval. When this closed interval is big enough, the probability mass on both ends will be very small. Thus in terms of “probability”, it is neglegible.

We will prove the third claim under the assumption the g (·) is monotonic increasing. For any t

Pr ( g (X_{n}) ≤ t) = Pr ( X_{n} ≤ g^{−1}(t))
→ Pr ( X ≤ g^{−1}(t)) = Pr ( g (X) ≤ t)

Thus

X_{n} →^{d} X ⇒ g (X_{n}) →^{d} g (X)

Theorem 10 If |X_{n} − Y_{n}| →^{p} 0 and Y_{n} →^{d} Y, then X_{n} →^{d} Y.

Theorem 11 If X_{n} →^{d} X and Y_{n} →^{p} 0, then X_{n}Y_{n} →^{d} 0.

Theorem 12 If X_{n} →^{d} X and Y_{n} →^{p} c (constant), then (a) X_{n}+Y_{n} →^{d} X +c;

(b) X_{n}Y_{n} →^{d} Xc; (c) X_{n}/Y_{n} →^{d} X/c.

This last theorem can be used to find the limiting distribution of a function of a r.v. whose limiting distribution is known.

Corollary 13 Let α_{n} be a sequence of nonstochastic nunbers tending to infinity, θ a fixed number, and α_{n} (Z_{n} − θ) →^{d} X. Let g be a function whose first-order derivatives are continuous. Then

α_{n} (g (Z_{n}) − g (θ)) →^{d} g′ (θ) X

Proof: (Bickel and Doksum) Note that

Z_{n} − θ = \frac{1}{α_{n}} [α_{n} (Z_{n} − θ)] →^{d} 0 · X →^{d} 0

Thus Z_{n} →^{p} θ. Next apply the mean value theorem

α_{n} (g (Z_{n}) − g (θ)) = g′ (θ^{∗}) [α_{n} (Z_{n} − θ)]

for θ^{∗} between Z_{n} and θ. Because θ^{∗} →^{p} θ and g′ (·) is continouous, g′ (θ^{∗}) →^{p} g′ (θ). Also by assumption α_{n} (Z_{n} − θ) →^{d} X. Thus

α_{n} (g (Z_{n}) − g (θ)) →^{d} g′ (θ) X

1.2 The Law of Large Numbers

1.2.1 Weak Law of Large Numbers

Theorem 14 (Chebychev) Let {X_{n}, n ∈ ℵ} be a sequence of independent r.v.’s such that EX_{i} = μ_{i}, var (X_{i}) = σ_{i}^{2} < c < ∞ for all n ∈ ℵ, then

lim_{n→∞} Pr (|\frac{1}{n} ∑_{i=1}^{n} X_{i} − \frac{1}{n} ∑_{i=1}^{n} μ_{i}| < ε) = 1

Proof: Since the X_{i}’s are independent

var (\frac{1}{n} ∑_{i=1}^{n} X_{i}) = \frac{1}{n^{2}} ∑_{i=1}^{n} σ_{i}^{2} ≤ \frac{c}{n}

By Chebyshev’s inequality

lim_{n→∞} Pr (|\frac{1}{n} ∑_{i=1}^{n} X_{i} − \frac{1}{n} ∑_{i=1}^{n} μ_{i}| ≥ ε) ≤ \frac{1}{ε^{2}} \frac{1}{n^{2}} ∑_{i=1}^{n} σ_{i}^{2} ≤ \frac{c}{nε^{2}} → 0

Theorem 15 (Markov) Let {X_{n}, n ∈ ℵ} be a sequence of r.v.’s such that

\frac{1}{n^{2}} var (∑_{i=1}^{n} X_{i}) → 0

then

lim_{n→∞} Pr (|\frac{1}{n} ∑_{i=1}^{n} X_{i} − \frac{1}{n} ∑_{i=1}^{n} μ_{i}| < ε) = 1

Proof: The same as the above theorem.

Remark 4 Note: If {X_{n}, n ∈ ℵ} is an independent sequence, then

\frac{1}{n^{2}} var (∑_{i=1}^{n} X_{i}) = \frac{1}{n^{2}} ∑_{i=1}^{n} var(X_{i}) → 0

This condition is called the Markov condition.

The above theorems impose restrictions on the variance of the random sequence. If {X_{n}, n ∈ ℵ} is an i.i.d. sequence, no restrictions on the variances are needed for the law of large numbers to hold.

Theorem 16 (Khinchin) Let {X_{n}, n ∈ ℵ} be a sequence of i.i.d. r.v.’s, then the existence of E (X_{i}) = μ for all i is sufficient to imply that for any ε > 0

lim_{n→∞} Pr (|\frac{1}{n} ∑_{i=1}^{n} X_{i} − μ| < ε) = 1

1.2.2 Strong Law of Large Numbers

Theorem 17 (Kolmogorov) Let {X_{n}, n ∈ ℵ} be a sequence of independent r.v.’s such that EX_{i} = μ_{i}, var (X_{i}) = σ_{i}^{2} for all n ∈ ℵ. Then if they satisfy the following condition

∑_{i=1}^{∞} \frac{1}{i^{2}} σ_{i}^{2} < ∞

we can deduce that

Pr ( lim_{n→∞} \frac{1}{n} ∑_{i=1}^{n} X_{i} = \frac{1}{n} ∑_{i=1}^{n} μ_{i}) = 1

Proof: Omitted.

Note: If we compare the Markov WLLN and the Kolmogorov SLLN, we can see that the differences are the two restrictions on the variances. We could deduce the Markov condition

\frac{1}{n^{2}} ∑_{i=1}^{n} σ_{i}^{2} → 0

from the Kolmogorov condition

∑_{i=1}^{∞} \frac{1}{i^{2}} σ_{i}^{2} < ∞

1.3 The Central Limit Theorem

Let {X_{n}, n ∈ ℵ} be a sequence of i.i.d. r.v.’s such that EX_{i} = μ, var (X_{i}) = σ^{2}. Let S_{n} = ∑_{i=1}^{n} X_{n}. The central limit theorem considers the limiting behavior of

Y_{n} = \frac{S_{n} − E (S_{n})}{\sqrt{var (S_{n})}} = \sqrt{n} ( \frac{\frac{1}{n} ∑_{i=1}^{n} X_{n} − μ}{σ}) = \frac{1}{\sqrt{n}} ∑_{i=1}^{n} (\frac{X_{n} − μ}{σ})

We will prove the simplest central limit theorem.

Theorem 18 (Lindeberg-Levy) Using F_{n} (y) to denote the distribution function of Y_{n}, we have Y_{n} →^{d} N (0, 1), i.e.

lim_{n→∞} F_{n} (y) = lim_{n→∞} P (Y_{n} ≤ y) = ∫_{−∞}^{y} \frac{1}{\sqrt{2π}} exp \{−\frac{1}{2}u^{2}\} du

Proof 1: We assume that the moment generating function for Y_{n} exists.

M_{Y_{n}}(t) = exp\{−\sqrt{n}μt/σ\}M_{S_{n}}(\frac{t}{σ\sqrt{n}})
= exp\{−\sqrt{n}μt/σ\}[M_{x}(\frac{t}{σ\sqrt{n}})]^{n}
ln M_{Y_{n}}(t) = −\sqrt{n}μt/σ + n ln M_{x}(\frac{t}{σ\sqrt{n}})

Next expand M_{x}(\frac{t}{σ\sqrt{n}}) as a power series in t to get

ln M_{Y_{n}}(t) = −\sqrt{n}μt/σ + n ln[1 + μ′_{1} \frac{t}{σ\sqrt{n}} + μ′_{2} \frac{t^{2}}{2σ^{2}n} + μ′_{3} \frac{t^{3}}{6σ^{3}n^{2}} + ...

Since for large n the expansion of ln(1 + x) is a power series in x, that is,ln(1 + x) = x − \frac{1}{2}x^{2} + \frac{1}{3}x^{3} − ⋯ so we can write

ln M_{Y_{n}}(t) = −\sqrt{n}μt/σ + n\{[μ′_{1} \frac{t}{σ\sqrt{n}} + μ′_{2} \frac{t^{2}}{2σ^{2}n} + μ′_{3} \frac{t^{3}}{6σ^{3}n\sqrt{n}} + ...]

−\frac{1}{2}[μ′_{1} \frac{t}{σ\sqrt{n}} + μ′_{2} \frac{t^{2}}{2σ^{2}n} + μ′_{3} \frac{t^{3}}{6σ^{3}n\sqrt{n}} + ...]^{2}

+\frac{1}{3}[μ′_{1} \frac{t}{σ\sqrt{n}} + μ′_{2} \frac{t^{2}}{2σ^{2}n} + μ′_{3} \frac{t^{3}}{6σ^{3}n\sqrt{n}} + ...]^{3} − ...\}

Collecting in powers of t we get:

ln M_{Y_{n}}(t) = (−\sqrt{n}μ/σ + \sqrt{n}μ′_{1}/σ)t + (μ′_{2}/2σ^{2} − μ_{1}^{′2}/2σ^{2})t^{2}
+(μ′_{3}/6σ^{3}\sqrt{n} − μ′_{1}μ′_{2}/2σ^{3}\sqrt{n} + μ_{1}^{′3}/3σ^{3}\sqrt{n})t^{3} + ...

But μ′_{1} = μ and μ′_{2} − μ_{1}^{′2} = σ^{2}. Therefore

ln M_{Y_{n}}(t) = \frac{1}{2}t^{2} + (μ′_{3}/6 − μ′_{1}μ′_{2}/2 + μ_{1}^{′3}/3)t^{3}/σ^{3}\sqrt{n} + ...

But the coefficient of t^{r} is a constant time 1/\sqrt{n^{r−2}} so that

lim_{t→∞} ln M_{Y_{n}}(t) = \frac{1}{2}t^{2}

and thus

lim_{t→∞} M_{Y_{n}}(t) = exp\{\frac{1}{2}t^{2}\}

which is the moment generating function for the N(0, 1).

Proof 2: We can also use the 1-1 correspondence theorem on the characteristic function and the distribution function, i.e., we will prove that the characteristic function of Y_{n}, ϕ_{n} (t), converges to the characteristic function of N (0, 1), exp{−1/2t^{2}}.

Denote (X_{n} − μ) /σ by ξ_{n}. Then ξ_{n} is also i.i.d. with mean 0, variance 1, and characteristic function φ (t) where φ (0) = 1, φ′ (0) = 0 and φ′′ (0) = 1. We have the following expansion for φ (t),

φ (t) = 1 − \frac{t^{2}}{2} + ct^{3} + o (t^{3})

where c = −i ∫ x^{3}f (x) dx. Thus the characteristic function of Y_{n}, φ_{n} (t), is

φ_{n} (t) = [1 − \frac{t^{2}}{2n} + \frac{ct^{3}}{n^{3/2}} + o (\frac{t^{3}}{n^{3/2}} )]^{n} → exp (−1/2t^{2})

Let f (n) = (ct^{3} + o (t^{3})) /\sqrt{n} in the following theorem, the above limiting result is obtained. The limit is nothing but the characteristic function of the standard normal distribution N (0, 1).

The last convergence depends on the following theorem:

Theorem 19 If lim_{n→∞} f (n) = 0, then

lim_{n→∞} [1 + \frac{b}{n} + \frac{f (n)}{n}]^{n} = lim_{n→∞} [1 + \frac{b}{n}]^{n} = e^{b}

Proof:

n log [1 + \frac{b}{n} + \frac{f (n)}{n}]

= n {log [n + b + f (n)] − log n}
= n ∫_{n}^{n+b+f(n)} \frac{1}{x}dx =^{mean value theorem} \frac{n}{n + θb + θf (n)} [b + f (n)]

for some θ ∈ (0, 1). Because

lim_{n→∞} \frac{θb + θf (n)}{n} = 0

Thus

lim_{n→∞} n log [1 + \frac{b}{n} + \frac{f (n)}{n}]

= lim_{n→∞} \frac{n}{n + θb + θf (n)} [b + f (n)] = b

So

lim_{n→∞} [1 + \frac{b}{n} + \frac{f (n)}{n}]^{n}
= lim_{n→∞} exp \{log [1 + \frac{b}{n} + \frac{f (n)}{n}]^{n}\}
= exp \{ lim_{n→∞} n log [1 + \frac{b}{n} + \frac{f (n)}{n}]\}
= e^{b}

Theorem 20 (Linderberg-Feller) Let {X_{n}, n ∈ ℵ} be a sequence of independent r.v.’s such that EX_{i} = μ_{i} and var (X_{i}) = σ_{i}^{2} < ∞. Then the relations (a)

lim_{n→∞} max_{1≤i≤n} \frac{σ_{i}}{c_{n}} = 0, where c_{n} = (∑_{i=1}^{n} σ_{i}^{2})^{1/2}

and (b)

lim_{n→∞} F_{n} (y) = lim_{n→∞} P (Y_{n} ≤ y) = ∫_{−∞}^{y} \frac{1}{\sqrt{2π}} exp \{−\frac{1}{2}u^{2}\} du

holds, if and only if,

lim_{n→∞} \frac{1}{c_{n}^{2}} ∑_{i=1}^{n} ∫_{|x−μ_{i}|>εc_{n}} (x − μ_{i})^{2} dF_{i} (x) = 0 for all ε > 0

The above condition is to control the tail behavior of all the random variables {X_{n}, n ∈ ℵ}. Let’s do the following deduction

\frac{1}{c_{n}^{2}} ∑_{i=1}^{n} ∫_{|x−μ_{i}|>εc_{n}} (x − μ_{i})^{2} dF_{i} (x)

≥ ε^{2} ∑_{i=1}^{n} Pr (|X_{i} − μ_{i}| > εc_{n})
≥ ε^{2} max_{1≤i≤n} Pr (|X_{i} − μ_{i}| > εc_{n})

i.e.

max_{1≤i≤n} Pr (|X_{i} − μ_{i}| > εc_{n})
≤ \frac{1}{ε^{2}c_{n}^{2}} ∑_{i=1}^{n} ∫_{|x−μ_{i}|>εc_{n}} (x − μ_{i})^{2} dF_{i} (x)

Thus the largest tail of all the variables X_{n}, n ∈ ℵ goes to zero. This shows that the heart of the CLT is the condition that no one r.v. dominates the sequence of sums, that is, each (X_{i} − μ_{i}) /σ_{i} is smaller relative to the sum [S_{n} − E (S_{n})] /c_{n} as n increases. Hence the CLT refers to the distributional behavior of the summation of an increasing number of r.v.’s which individually do not exert any significant effect on the behavior of the sum.

Lindeberg-Feller CLT is the most general CLT about the independent random sequences. The following theorem is a special case of the Lindeberg-Feller theorem.

Theorem 21 Let {X_{n}, n ∈ ℵ} be a sequence of independent r.v.’s such that EX_{i} = μ_{i}, var (X_{i}) = σ_{i}^{2} < ∞ and E (|X_{i}|^{2+δ}) < ∞ for δ > 0, then if

lim_{n→∞} \frac{1}{c_{n}^{2+δ}} ∑_{i=1}^{n} E (|X_{i} − μ_{i}|^{2+δ}) = 0

holds, the CLT holds, i.e.

lim_{n→∞} F_{n} (y) = lim_{n→∞} P (Y_{n} ≤ y) = ∫_{−∞}^{y} \frac{1}{\sqrt{2π}} exp \{−\frac{1}{2}u^{2}\} du

Proof: We will prove that the Lindeberg-Feller condition holds, i.e.

lim_{n→∞} \frac{1}{c_{n}^{2}} ∑_{i=1}^{n} ∫_{|x−μ_{i}|>εc_{n}} (x − μ_{i})^{2} dF_{i} (x) = 0 for all ε > 0

lim_{n→∞} \frac{1}{c_{n}^{2}} ∑_{i=1}^{n} ∫_{|x−μ_{i}|>εc_{n}} (x − μ_{i})^{2} dF_{i} (x)

≤ lim_{n→∞} \frac{1}{c_{n}^{2+δ}} ∑_{i=1}^{n} ∫_{|x−μ_{i}|>εc_{n}} c_{n}^{δ} (x − μ_{i})^{2} dF_{i} (x)
≤ lim_{n→∞} \frac{1}{c_{n}^{2+δ}} ∑_{i=1}^{n} ∫_{|x−μ_{i}|>εc_{n}} \frac{1}{ε^{δ}} |x − μ_{i}|^{2+δ} dF_{i} (x)
≤ lim_{n→∞} \frac{1}{ε^{δ}} \frac{1}{c_{n}^{2+δ}} ∑_{i=1}^{n} ∫ |x − μ_{i}|^{2+δ} dF_{i} (x)
= \frac{1}{ε^{δ}} lim_{n→∞} \frac{1}{c_{n}^{2+δ}} ∑_{i=1}^{n} E (|X_{i} − μ_{i}|^{2+δ}) = 0

1.4 The ‘big O’ and ‘little o’ notation

Definition 22 The sequence {a_{n}, n ∈ ℵ} is said to be ‘at most of order b_{n}’ and denoted by a_{n} = O (b_{n}) as n → ∞ if

lim_{n→∞} (\frac{|a_{n}|}{b_{n}}) < K

for some constant K > 0.

Definition 23 The sequence {a_{n}, n ∈ ℵ} is said to be ‘of smaller order than b_{n}’ and denoted by a_{n} = o (b_{n}) as n → ∞ if

lim_{n→∞} (\frac{a_{n}}{b_{n}} ) = 0

Example:

(\frac{1}{2n^{2} − 3}) = O (\frac{1}{n^{2}} ); (n + 1) = O (n) = o (n^{2}); exp \{−n\} = o (n^{−δ}) for any δ > 0

(\frac{2n + n^{2}}{5n^{2} + n^{3}} ) = O (n^{−1}); log_{e}n = o (n^{α}) ; (6n^{2} + 3n) = o (n^{3}) = O (n^{2})

The O, o notation considered above can be extended to the case of stochastic convergence, convergence almost surely and in probability.

Definition 24 Let {X_{n}, n ∈ ℵ} be a sequence of r.v.’s and sequence {c_{n}, n ∈ ℵ} a sequence of positive real numbers. We say that (i) X_{n} is at most of order c_{n} in probability if there exists nonstochastic sequence {a_{n}, n ∈ ℵ} such that

a_{n} = O (1) and (\frac{X_{n}}{c_{n}} − a_{n}) →^{p} 0

(ii) X_{n} is of order smaller than c_{n} in probability, denoted by X_{n} = o_{p} (c_{n}), if

(\frac{X_{n}}{c_{n}} ) →^{p} 0

Example: Let {X_{n}, n ∈ ℵ} be a sequence of i.i.d. r.v.’s such that EX_{i} = μ ≠ 0, var (X_{i}) = σ^{2},.Then

\frac{1}{n} ∑_{i=1}^{n} X_{i} = O_{p} (1)

\frac{1}{n} ∑_{i=1}^{n} (X_{i} − μ) = o_{p} (1) and \frac{1}{n} ∑_{i=1}^{n} (X_{i} − μ) = O_{p} (\frac{1}{\sqrt{n}})