B9323 Introduction to Econometrics and Statistical Inference

B9323 Introduction to Econometrics and

Statistical Inference
Fanyin Zheng
Division of Decision, Risk, and Operations

Fall 2016

B9323 Introduction to Econometrics and Statistical Inference

Outline

Estimator and finite sample properties

Inequalities and convergence

LLN, CLT, and delta method

Confidence interval

B9323 Introduction to Econometrics and Statistical Inference

Estimator

X1 Xn are independent identically distributed (iid) with

density function f (x, )

Estimate = (, 2 ), = E(Xi ), 2 = V(Xi )

B9323 Introduction to Econometrics and Statistical Inference

Estimator

X1 Xn are independent identically distributed (iid) with

density function f (x, )

Estimate = (, 2 ), = E(Xi ), 2 = V(Xi )

1
n

Pn

i=1 Xi

= X

B9323 Introduction to Econometrics and Statistical Inference

Estimator

X1 Xn are independent identically distributed (iid) with

density function f (x, )

Estimate = (, 2 ), = E(Xi ), 2 = V(Xi )

2 =

1
n

Pn

i=1
1 Pn

Xi = X

i=1 (Xi

X )2

B9323 Introduction to Econometrics and Statistical Inference

Unbiased estimator

is unbiased if E =

B9323 Introduction to Econometrics and Statistical Inference

Unbiased estimator

I
I

is unbiased if E =
P 
P
i Xi
E(
) = E
= n1 i E(Xi ) =
n

B9323 Introduction to Econometrics and Statistical Inference

Unbiased estimator

I
I
I

is unbiased if E =
P 
P
i Xi
E(
) = E
= n1 i E(Xi ) =
n
P
2
E(
2 ) = n1 E( i Xi2 nX 2 ) = n1
n

(?)

B9323 Introduction to Econometrics and Statistical Inference

Unbiased estimator

I
I
I
I

is unbiased if E =
P 
P
i Xi
E(
) = E
= n1 i E(Xi ) =
n
P
2
E(
2 ) = n1 E( i Xi2 nX 2 ) = n1
n
P
n
1
2
Make it unbiased
2 = n1
i=1 (Xi X )

(?)

B9323 Introduction to Econometrics and Statistical Inference

Standard error

I
I

and
2 are two random variables
Standard
error of an estimator is its standard deviation
q

V()
p
pP

= V(X ) =
V(Xi )/n2 = / n
For example, s.e.()

B9323 Introduction to Econometrics and Statistical Inference

Efficiency

Let 1 , 2 be two estimators of . If E1 = E2 = ,

V(1 ) < V(2 ), then 1 is more efficient than 2 .

B9323 Introduction to Econometrics and Statistical Inference

Mean Squared Error

One way to compare estimators that are not necessarily

unbiased is to compute the mean squared error (MSE) of the
estimators
= E(( )2 )
If is an estimator of , then MSE()

= Var()
+ (Bias())
2
MSE()

(Example)

B9323 Introduction to Econometrics and Statistical Inference

Inequalities

EX
x

Chebyshev Inequality: if X > 0, P(X > x)

Proof

Or P(|X | > )

Markov Inequality: if Ef (x) < and f is nonnegative and

monotonically increasing, 0, and f () > 0,
|)
P(|X | > ) Ef (|X
f ()

VX
,
2

where > 0

B9323 Introduction to Econometrics and Statistical Inference

Lets apply the inequalities

V(X )
2
=
,
2
n2
2

where n
2 0, as n . Sample mean X becomes more and
more concentrated around the true parameter as sample size
increases.
P(|X | > )

B9323 Introduction to Econometrics and Statistical Inference

Law of Large Number

Suppose Xi P
are iid random variables with EXi = and EXi2 < .
1

Let Xn = n ni=1 Xi . Then > 0, P(|Xn | > ) 0, as

n . I.e. Xn is consistent for .

10

B9323 Introduction to Econometrics and Statistical Inference

Convergence in probability

Xn
X or Xn converges in probability to X R, if > 0,
P(|Xn X | > ) 0, as n .

B9323 Introduction to Econometrics and Statistical Inference

Convergence in probability

Xn
X or Xn converges in probability to X R, if > 0,
P(|Xn X | > ) 0, as n .
Example I: X1 , ..., Xn are iid, EXi2 = , EXi4 < . Let Vi = Xi2 .
P
p
Then n1 ni=1 Vi
.

11

B9323 Introduction to Econometrics and Statistical Inference

Convergence in probability

Example II: is
2 a consistent estimator of 2 ?

B9323 Introduction to Econometrics and Statistical Inference

Convergence in probability

Example II: is
2 a consistent estimator of 2 ?
P
I
2 = n1 i Xi2 Xn2
P 2 p
I 1
EXi2 , if EXi4 <
i Xi
n
P
p
I 1
EXi
i Xi
n
P 2
p
1P
2
I Need to show 1
EXi2 (EXi )2
i Xi ( n
i Xi )
n

B9323 Introduction to Econometrics and Statistical Inference

Properties of

Y = Xn + Yn
X +Y
X , Yn
Xn

Xn
X = cXn
cX c R

Xn
X , g is continuous = g (Xn )
g (X )

B9323 Introduction to Econometrics and Statistical Inference

Properties of

Y = Xn + Yn
X +Y
X , Yn
Xn

Xn
X = cXn
cX c R

Xn
X , g is continuous = g (Xn )
g (X )

Proof of (2): P(|cXn cX | > ) = P(|Xn X | >

as n .

|c| )

0,

B9323 Introduction to Econometrics and Statistical Inference

op and Op notation

Xn p

bn

0, as n 0

Xn = op (bn ) if

Example: Xn = op (1) Xn
0

Xn = Op (bn ) if > 0, M, s.t. P(| Xbnn | > M) < , n

Example: Xn = Op (1) ?

Xn = op (an ), Yn = op (bn ) = Xn Yn = op (an bn )

14

B9323 Introduction to Econometrics and Statistical Inference

Convergence in distribution

Xn converges to a random variable X in distribution, i.e. Xn

X,
if Fn (x) F (x) at all continuous points of F (x).

15

B9323 Introduction to Econometrics and Statistical Inference

Convergence in distribution

Xn converges to a random variable X in distribution, i.e. Xn

X,
if Fn (x) F (x) at all continuous points of F (x).


0 x<
1 x
P(X = 0) = 1, then Fn (x) F (x), x > 0, as n .
But F (x) = 1 6= limn Fn (0) = 0.
Suppose P(Xn =

1
n)

= 1, Fn (x) = P(Xn x) =

1
n
1
n

B9323 Introduction to Econometrics and Statistical Inference

Convergence in distribution: moment generating function

If 1) gn (t) = Eexp(tXn ) t R is the moment generating function
of Xn , 2) X is a random variable with mgf g (t) = Eexp(tX )
d

t R, 3) gn (t) g (t) t R as n . Then Xn

X.

B9323 Introduction to Econometrics and Statistical Inference

Convergence in distribution: moment generating function

If 1) gn (t) = Eexp(tXn ) t R is the moment generating function
of Xn , 2) X is a random variable with mgf g (t) = Eexp(tX )
d

t R, 3) gn (t) g (t) t R as n . Then Xn

X.

Example: Xn =
p
X
0.

1
1

n(Xi EXi ) d

?
VXi

1
2
1
2

, then EXi = 0, VXi = 1. By LLN,

B9323 Introduction to Econometrics and Statistical Inference

Convergence in distribution: moment generating function

If 1) gn (t) = Eexp(tXn ) t R is the moment generating function
of Xn , 2) X is a random variable with mgf g (t) = Eexp(tX )
d

t R, 3) gn (t) g (t) t R as n . Then Xn

X.


1
1

Example: Xn =
p
X
0.

n(Xi EXi ) d

?
VXi

gn (t) = Eexp(t
= (1 +

t2
2n

Xi

i n)

o( n1 ))n

1
2
1
2

, then EXi = 0, VXi = 1. By LLN,

tXi

i Eexp( n )

2
exp( t2 ),


=

as n .

t
exp( t n )+exp(
)
n

n

(?)

B9323 Introduction to Econometrics and Statistical Inference

Central Limit Theorem

If Xi iid, EXi = , VXi = 2 , then

n(X ) d

N(0, 1).

B9323 Introduction to Econometrics and Statistical Inference

Central Limit Theorem

If Xi iid, EXi = , VXi = 2 , then

n(X ) d

N(0, 1).

Proof: Let Yi = Xi , then EYi = 0, VYi = 1, and

nY = n(X )/
Phas mgf
Q

MnY (t) = Eexp(t i Yi / n) = i MYi (t/ n)


n
t2
= MYi (0) + MY0 i (0) tn + MY00i (0) 2n
+ o( n1 )
2

t
+ o( n1 ))n
= (1 + 2n
2
exp( t2 ), as n

B9323 Introduction to Econometrics and Statistical Inference

Slutskys Theorem

If Xn
X , Yn
c R, then
d

Xn + Yn
X +c

Xn Yn
cX

Xn /Yn
X /c

g (Xn , Yn )
g (X , c), g is a continuous function
(Continuous Mapping Theorem)

B9323 Introduction to Econometrics and Statistical Inference

Delta Method

d
If n(Xn )
N(0, 2 ), then

d
n(g (Xn ) g ())
N(0, [g 0 ()]2 2 ).

19

B9323 Introduction to Econometrics and Statistical Inference

Delta Method

d
If n(Xn )
N(0, 2 ), then

d
n(g (Xn ) g ())
N(0, [g 0 ()]2 2 ).
Proof: g (Xn ) = g () + g 0 ()(Xn ) + o(Xn )

= n[g (Xn ) g ()] = g 0 () n(Xn ) + o( n(Xn ))

d
1. g 0 () n(Xn )
N(0, g 0 ()2 2 )

2. n(Xn ) = Op (1) = o( n(Xn )) = op (1)

(?)

19

B9323 Introduction to Econometrics and Statistical Inference

Delta Method

d
N(0, ), where is a
Let Xn be a k 1 vector, n(Xn )
k 1 vector, is a k k matrix, g : Rk Rm , 
then

T 


d
g ()
g ()
n(g (Xn ) g ())
N 0, x
x

20

B9323 Introduction to Econometrics and Statistical Inference

Remarks

Unbiasedness n = ? Consistency

= ? Op (1)

CLT = ? LLN

= ?

21

B9323 Introduction to Econometrics and Statistical Inference

Confidence Interval

A confidence interval C (X) for a parameter is a set of

possible values which contains with some specified
probability

For example,
C (X) = [X cn , X + cn ]
is (1 ) confidence interval for , where cn is chosen s.t.
P( C (X)) = 1

B9323 Introduction to Econometrics and Statistical Inference

Confidence Interval

Let X be the estimator for , construct confidence interval

under the following assumptions:

Case I: let Xi iid N(, 2 ), where is known

Case II: Xi iid N(, 2 ), where is unknown

Case III: Xi iid unknown distribution, EX 2 <

23