Multiple Regression Analysis

OLS Asymptotics
Ch. 5
u X y + =
] x x i X
k 1
[ =
0 ) ( ) | ( = = X | x x
k 1
u E u E
Earlier we covered finite sample (or exact) properties of
the OLS estimators. We had the following results:

Assumption MLR.1: the model of interest is linear in
parameters,
Assumption MLR.2: Our data are a random sample of n
observations
Assumption MLR.3: Matrix of regressors
has full column rank
Assumption MLR.4: Zero conditional mean
Given assumptions MLR1-4, OLS estimator is unbiased:
E(b|X)=. This is an exact property, because it holds for
any sample size (including small).
2 2
) | ( ) | ( o = = X X
i i
u E u V
j i u u Cov
j i
= = , 0 ) | , ( X
Assumption MLR.5: Spherical disturbances

=> homoskedasticity

With this additional assumption, OLS can be shown
to be the best linear unbiased estimator (Gauss-
Markov theorem).

This is also an exact property of the OLS.
=> non-autocorrelation
Assumption MLR.6: conditional on X, errors are normally
distributed with mean 0 and constant variance
2
:
.
This assumption allows making a stronger case about
OLS estimator: it is the best among all (not just linear)
unbiased estimators.

As above, this is an exact property of the OLS.

In addition to making OLS the best unbiased estimator
this last assumption also allows us constructing the exact
sampling distribution of b which leads to the t and F
distribution for t and F statistics.
What will happens to our results if Assumption MLR.6
does not hold?

Fortunately, given all the other assumptions we made, OLS
has good large sample properties (or asymptotic
properties).

In particular, t and F statistic have approximately t and F
distribution, at least in large samples.
Consistency of the OLS

Sometimes an estimator fails to be unbiased. In those
cases it has to be at least consistent:
> n as b P
n
0 ) | (| c |
As sample size growths, the probability of being far from
the true parameter is approaching zero.

Consistency is a minimal requirement for an estimator.

Theorem: Under assumptions MLR1-4 OLS estimator is
unbiased (as we know) and also consistent.

1 1
2
1
1
1
2
1
1
1
) (
) , (
) (
1
) (
1
) (
) (
| | | | = +

+ =

=
=
=
=
x Var
u x Cov
x x
n
u x x
n
x x
y x x
i
n
i
i i
n
i
i
n
i
i i
n
i

0 ) , ( = u x Cov
For the case of simple regression model:

In this derivation we used the fact that
which is a weaker assumption than SLR4 (x and u are
independent).
(
(

|
.
|

\
|
+ =
(
(

|
.
|

\
|
+ =
|
.
|

\
|
+ = + = + = =

n n
p
n n
p p
n n
u X X X

u X X X
b
u X X X
u X X X u X X X X y X X X b
1 1
1
1 1 1
' '
lim
' '
lim ) lim(
' '
' ) ' ( ) ( ' ) ' ( ' ) ' (
consistent is OLS 0
'
lim If
'
lim
' '
lim ) lim(
'
that assume s Let'
1
n
lim
=
|
.
|

\
|
|
.
|

\
|
+ =
(
(

|
.
|

\
|
+ =
=
|
.
|

\
|

n
p
n
p
n n
p p
n
u X
u X
Q
u X X X
b
Q
X X
1
A little bit of matrix algebra to deal with the multiple linear
regression case:

= = = =
|
.
|

\
|
= = =
|
.
|

\
|
0 0 lim
'
) | ( '
1
|
'
V 2.
0 lim 0 ) | ( '
1
|
'
E 1.
2
2
Q
X X
X X u X X
u X
X u X X
u X
V
n n
V
n n
E E
n n
o
0
'
lim =
|
.
|

\
|
n
p
u X
0 Q
u X X X
b
1
= + =
(
(

|
.
|

\
|
+ =

1
' '
lim ) lim(
n n
p p
consistent is OLS
Previously we have seen that if
0 ) | ( = X u E
Previously we have seen that if , OLS
estimator is not unbiased.


Now we can add that if any of the independent variables
are correlated with the error term, the OLS will also be
inconsistent!


This is very bad: there is bias in small samples and it does
not go away as we increase sample size.
0 ) | ( = X u E

Hypothesis testing without MLR 6
Target: Assumption MLR 6

Assumption MLR 6 plays crucial role in hypothesis testing:






What happens to the distribution of t and F statistics when
the errors are not normal?

=

=

) 1 , (
2
) 1 (
2
) 1 (
2
) 1 ( 1
~
) 1 /(
/
~
) 1 /(
) (
) ] [ , ( ~ ) , ( ~
k n J
k n
J
k n
k n
j
j j
F
k n
J
F
t
k n
SN
b e s
b

_
_
_
|

X X' N b I 0 N u
2 2

If errors are not normally distributed, b-OLS will not be
normally distributed, and b\SE(b) will not have t-distribution
and F-statistic will not have exact F distribution.
Theorem: Even thought b-OLS is not normal if assumption
MLR6 does not hold, it is asymptotically normal =>
approximately normal in large enough samples.
Q
X X
u X X X
b u X X X b u X X X b
1
1 1

|
.
|

\
|
|
.
|

\
|
= = + =


n
n n
n
'
that assumption same the use We
' '
) ( ' ) ' ( ' ) ' (
) , (
'
2
Q 0
u X
o N
n
d

|
.
|

\
|
More matrix algebra:

Using the central limit theorem it can be shown that
.
) ] [ , ( ) ] [ , ( ) , (
1
) , ( ) , (
' '
) (
? ] ' [
1 2 1 2 1 2
1 2 2 1
1 2

=
= |
.
|

\
|
|
.
|

\
|
=
X X
1
Q b Q 0 Q 0 b
Q 0 Q 0 Q
u X X X
b
s
a a
d
n N n N N
n
N N
n
n
n
o o o
o o
2
o
2 2
) lim( o = s p
1
X X

) ' (
2
s
It can be shown that s2 is a consistent estimator of
(in addition to being unbiased. HW problem).
Also, since XX-> nQ, we can use
as an appropriate estimator of the covariance matrix.

as an
From these results it follows that
) 1 , 0 ( ) ( / ) ( )) var( , ( N b se b b N b
a
j j j j j
a
j

| |
Where as before:

And the R-squared is from regressing xj on all the other
independent variables. Now:
Take the square root out of the asymptotic variance of bj
Estimate with s
And we have the estimated asymptotic standard error of bj:
s(bj).

which can be used to do hypotheses testing. All our steps
will be as before, the difference is that now the results will
be asymptotic (valid for large samples).

=
=

=
n
i
j ij j
j j
j
x x SST
R SST
b Var
1
2
2
2
) ( ,
) 1 (
) (
o
From these results it follows that
) 1 , 0 ( ) ( / ) ( N se
d
j j j

| | |

(use t table for testing, since it shows critical values for
standard normal when n is large).

How about the F-statistic? It has an approximate F-
distribution in large samples. So, we use the same steps as
we did before to perform hypothesis testing.

To show the asymptotic normality of b-OLS we do need to
assume homoskedasticity (same error variance for all
observations).The heteroskedasticity case will be
discussed in our next topic.

Finally, it can be shown that under assumptions MLR 1-5
(Gauss-Markov assumptions) b-OLS is asymptotically
efficient among a certain class of estimators, B:




To claim the asymptotic normality and efficiency of b-
OLS we do need assumptions MLR1-5. In particular, we
NEED to assume homoskedasticity (same error variance
for all observations).

The case of heteroskedasticity is what we will discuss
next.
) ( ) ( B V Asymptotic b V Asymptotic
OLS
s
OLS Asymptotics
MLR 1,2,3,4 => b-OLS is unbiased AND consistent

MLR 1,2,3,4 + MLR 5 => b-OLS is BLUE

MLR 1,2,3,4 + MLR 5 + MLR 6 => b-OLS is BUE


b-OLS Unb, Lin Unbiased MLR 1-5 MLR 1-6
B1 Unb, Lin Unb, Lin b-OLS b-OLS
B2 Unb, Lin Unb, Lin b-OLS b-OLS
B3 Bsd, Nlin
B4 Unb, Nlin Unb, Nlin b-OLS
B5 Unb, Lin Unb, Lin b-OLS b-OLS
B6 Bsd, Lin
B7 Unb, Nlin Unb, Nlin b-OLS