Review on Linear Algebra (ECO327)

Yuanyuan Wan
This is note is prepared for ECO327. The materials in this lecture, except for those marked as
optional, will be used in later lectures.
1 Matrices
1.1 Notations
Denition 1 An n m matrix is an array of numbers:
A =

a
11
a
12
a
1m
a
21
a
22
a
2m
.
.
.
.
.
.
.
.
.
.
.
.
a
n1
a
n2
a
nm

nm
= (a
ij
)
nm
The matrix whose all entries are 0 is called zero matrix, denoted by O.
Denition 2 A vector is an ordered set of numbers arranged either in a row or a column. A
column vector is a matrix with one column. A row vector is a matrix with one row.
1
Remark 1 Without special notication, all vectors appear in this course are assumed to be column
vector.
Remark 2 Note that matrix A in Denition 1 is a collection of n row vectors, or a collection of
m column vectors.
Remark 3 A 21 vector x is often used to denote the coordinates of a point in a plane. The rst
number corresponds to the horizontal axis and the second one corresponds to the vertical axis. We
often write x R
2
.
Remark 4 More generally, we often write an n 1 vector as x R
n
. It gives the coordinate of a
point in a ndimensional space.
A special kind of matrix is square matrix, which has same number of columns and rows, i.e.
n = m. Here is some special square matrices:
Symmetric Matrix: a square matrix in which a
ij
= a
ji
for all i and j.
Diagonal Matrix: a square matrix in which a
ij
= 0 if i = j, i.e. all o-diagonal entries are zero.
Scalar Matrix: a diagonal matrix in which a
ii
= for all i, i.e., all diagonal entries are the same.
1
In this notes, except for special notication, I use small letter (e.g. a, b, ) to denote a real number, use bold small
letter (e.g. a, b, x, i) to denote a vector, and use capital letter (e.g. A, B, Q) to denote a matrix. But sometimes, I
use bold capital letter to denote some special matrix (e.g. M0, O, I).
1
Identity Matrix I: a scalar matrix in which a
ii
= 1 for all i.
Triangular Matrix: a square matrix in which a
ij
= 0 for all i < j, or a
ij
= 0 for all i > j. The
former is called lower triangular matrix, and the latter is called upper triangular matrix.
A =

1 3 3 5
0 6 8 0
0 0 9 3
0 0 0 1

1.2 Operations
We can dene the following operation on matrix:
1. Equality: let A and B be two n m matrices, then A = B i for all i and j,
a
ij
= b
ij
2. Matrix addition: let A and B be two n m matrices, then:
A + B = (a
ij
) + (b
ij
) (a
ij
+ b
ij
)
nm
Remark 5 Matrix addition is well dened only if the two matrix have same number of
columns and rows.
3. Scalar multiplication: let A be a n m matrix, and be a real number, then:
A = (a
ij
) ( a
ij
)
4. Matrix multiplication: let A be a n m matrix, B be a ml matrix, then:
AB = (a
ij
) (b
ij
)

k=1
a
ik
b
kj

nl
Remark 6 Matrix Multiplication is well dened only if the number of columns of the rst
matrix is the same as the number of rows of the second matrix. For example, if l = n,
then B A is not dened. Generally, A B = B A. To distinguish them, we dene
premultiplication and postmultiplication of matrix. In the case of A B, we say B is
premultiplied by A, or A is postmultiplied by B.
Remark 7 Suppose A, B, and C are matrices such that the operations are well dened, we
have the following properties:
A + B = B + A
(A + B) + C = A + (B + C)
A(B + C) = AB + AC
A(BC) = (AB) C
2
a(AB) = (aA)B = A(aB).
AI = A, IA = A.
AO = O, OA = O.
5. Transpose: if A is an n m matrix, then the transpose of A, A
T
(or A

), is the m n
matrix with the rows and columns switched:
A

(or A
T
) =

a
11
a
21
a
n1
a
12
a
22
a
n2
.
.
.
.
.
.
.
.
.
.
.
.
a
1m
a
2m
a
nm

mn
Notice that A = (A

. Also notice that if a is a row vector, then a

is a column vector, and

vice versa.
Remark 8 From the denition of symmetric matrix, we know:
A = A

i A is symmetric matrix
Remark 9 Let A, B, and C be three matrices, then:
(ABC)

= C

6. Trace of a Matrix
Denition 3 The trace of a square matrix, denoted by tr (A), is given by tr (A) =

n
i=1
a
ii
.
The trace of a square matrix has the following property:
tr (A + B) = tr (A) + tr (B)
tr (k A) = k tr (A)
tr (A

) = tr (A)
tr (AB) = tr (BA)
tr (ABC) = tr (BCA) = tr (CAB)
tr

M
1
AM

= tr (A).
7. Inner product of vector: if a and b are two vectors with same length m, then
a b =
m

k=1
a
k
b
k
is called inner product of those two vectors.
In particular, if a and b are both row vectors, then
a b = ab

If a and b are both column vectors, then

a b = a

b
3
Example 1 Let {a
i
}
n
i=1
be n real numbers. Let i be a column vector of length n with all elements
equal to 1. Then the sum of the sequence of real numbers can be represented by:
n

i=1
a
i
= i

a
where a is a column vector with the l

th element equaling to a
l
.
The mean of the sequence, denoted by a, can be represented by:
a =
1
n
i

a
A special vector ( a, a, , a)

can be written as:

a
a
.
.
.
a

= i a =
1
n
ii

a
and
a

a
a
.
.
.
a

a
1
a
a
2
a
.
.
.
a
n
a

I
1
n
ii

a M
0
a
Denition 4 Let M be a square matrix. M is called idempotent matrix if M
2
= M. If M is a
symmetric idempotent matrix, then M

M=M.
In the above example, M
0
is a symmetric idempotent matrix. We will use M
0
a lot in our
course a lot. We will come back to idempotent matrices after introducing the notion of eigenvalues.
2 Linear dependence and ranks
Let x and y be two ndimensional vectors and a, b are two numbers. Then ax+by is called linear
combination of x and y.
Denition 5 A set of k vectors {x
1
, x
2
, , x
k
} in R
n
is linearly independent if and only if
none of those vectors can be written as the linear combination of other vectors in the set. If the
set of vector is not linear independent, then it is linear dependent.
The concept of linear independent can also be expressed as follows:
Denition 6 A set of vectors {x
1
, x
2
, , x
k
} in R
n
is linearly independent if and only if the
only set of real numbers which satises the following equation

1
a
1
+
2
a
2
+ +
k
a
k
= 0
is

1
=
2
= =
k
= 0
4
Example 2 The following two vectors in R
3
are linearly independent.
x
1
=

1
0
0

x
2
=

0
1
0

However, x
4
, x
5
and x
6
are not.
x
4
=

2
1
3

x
5
=

0
2
2

x
6
=

4
0
4

Given the above denitions, we are ready to dene the rank of matrix by steps.
Denition 7 Column and Row Rank of a Matrix. The number of linearly independent
columns vectors in a matrix A is called the column rank of the matrix, denoted by
c
(A) Similarly,
the number of linearly independent row vectors is called the row rank of the matrix, denoted by

r
(A).
By denition, if the matrix is n m, then
c
(A) m,
r
(A) n. If
c
(A) = m, then we say
the matrix has full column rank.
Theorem 1 The column rank and the row rank of a matrix are equal.
Since the column rank and row rank of a matrix are the same, we can dene rank of matrix
(A) as:
Denition 8 The rank of a matrix equals to its column rank.
It is clear that the rank of n m matrix satises:
(A) min (n, m)
If a matrix has full column rank, then we say that the matrix has full rank.
Example 3 The matrix formed by x
4
, x
5
and x
6
does not have full rank.
By the denition of rank of matrix, we can also have the following properties:
1. (A) = (A

)
2. Let B be a matrix constructed by interchanging any two rows or (columns) of A, then (A) =
(B)
3. Let B be a matrix constructed by multiplying a row (or column) with an non-zero real number
, then (A) = (B)
4. Let B be a matrix constructed by replacing some row (or column) with itself plusing another
row (or column), then (A) = (B)
5
Remark 10 The above three kinds of transformation is called Elementary row (column)
operation of matrices. Elementary operation will not change the rank of a matrix.
5. Let A be an n m matrix, B be an ml matrix, then (AB) min { (A) , (B)}.
6. Let A be an nm matrix, B be an mm matrix with full rank, C be an nn matrix with
full rank, then (AB) = (CA) = (A).
Example 4 (Optional) Find the rank of a matrix.
Since the rank is preserved under elementary operation, we can use elementary operation to
nd the rank of matrix. The target is to reduce the matrix to a diagonal one.
For example, let:
A =

1 3 4 5
2 2 7 9
3 3 9 12

Step 1, Times the rst row with 2, and add it to second row, we get:
A
(1)
=

1 3 4 5
0 4 1 1
3 3 9 12

Similarly, times the rst row of A

(1)
with 3, and add it to the third row, we get:
A
(2)
=

1 3 4 5
0 4 1 1
0 12 3 3

Step 2, then times the rst column of A

(2)
with 3, 4, and 5 respectively, and add them to
column 2, 3, and 4 of A
(2)
respectively, we get:
A
(3)
=

1 0 0 0
0 4 1 1
0 12 3 3

Notice that the rst column and row of A has been transformed into zeros except for a
11
. Now
do the same thing to the submatrix

4 1 1
12 3 3

At last, the original matrix will be transformed into:

A
(4)
=

1 0 0 0
0 1 0 0
0 0 0 0

It is obvious that the column rank of A

(4)
is 2, so is the rank of A
(4)
. However, A
(4)
is constructed
from A by applying elementary operation, hence

A
(4)

= (A) = 2.
6
3 Determinant of matrices
The notion of determinant of matrices only applies to square matrices. So in this section,
without special notication, all matrices under discussion are square matrices.
Denition 9 If A be a 1 1 matrix, (a), then the determinant of A is: |A| = a.
Denition 10 Let Mbe the set of square matrices with n columns. Then the determinant of the
square matrix is function f : MR given as follows:
f (A) |A| =

n
k=1
a
ik
(1)
i+k
|A
ik
| if n 2
a if n = 1
where A
ik
is obtained by deleting the i
th
row and k
th
column from A, and |A
ik
| is called a
minor of A, and (1)
i+k
|A
ik
| is called a cofactor of a
ik
.
As for the determinant of a matrix, we have the following properties:
1. Let D be a diagonal matrix
D =

a
11
0 0
0 a
22
0
.
.
.
.
.
.
.
.
.
.
.
.
0 0 a
mm

Then
|D| =
m

i=1
a
ii
2. Let A a diagonal matrices, then:
|AB| = |A| |B|
3. Let A be an n n matrix, be a real number, then:
|A| =
n
|A|
4.
|A| =

Denition 11 A square matrix is called nonsingular if |A| = 0, otherwise it is called singular.

Theorem 2 A square matrix is nonsingular if and only if it has full rank.
7
4 Inverse of matrices
Denition 12 Let A be a n n square matrix. An n n matrix B is called the inverse matrix
of A if it satises AB = BA = I, denoted by A
1
. An matrix is called invertible if it has a inverse
matrix.
Theorem 3 If a square matrix A has inverse matrix A
1
, then it is unique.
Denition 13 Let A be a nonsingular square matrix, and |A
ik
| be its minors and C
ik
be its
cofactors. Then the matrix A

is called the adjoint matrix of A if it is:

A

C
11
C
21
C
n1
C
12
C
22
C
n2
.
.
.
.
.
.
.
.
.
.
.
.
C
1n
C
2n
C
nn

nn
Remark 11 Please pay attention to the position of C
ik
in the matrix A

. We need to put it to
the k
th
row and j
th
column.
Theorem 4 A square matrix A is invertible i A is nonsingular. If it is invertible, then:
A
1
=
1
|A|
A

We have the following property for the inverse of matrix:

1. If A is invertible, then A
1
is also invertible and

A
1

1
= A.
2. (AB)
1
= B
1
A
1
.
3. If A is invertible, then A

is invertible and (A

)
1
=

A
1

.
4.

A
1

=
1
|A|
5 Partitioned Matrix
In practice, sometimes it is convenient to write a matrix into a partitioned form.
A =

1 | 3 4
|
2 | 2 7
3 | 3 12

A
11
A
12
A
21
A
22

A special case is blockdiagonal form.

A =

A
11
O
O A
22

8
The operations of partitioned matrix are of no dierence from before, provided the operation
of submatrices does not obey the rules.

A
11
O
O A
22

A
11
O
O A
22

11
A
11
O
O A

22
A
22

A
1
A
2

A
1
A
2

1
A
1
+ A

2
A
2

The inverse of an invertible blockdiagonal matrix is

A
11
O
O A
22

1
=

A
1
11
O
O A
1
22

The inverse of an invertible matrix is

A
11
A
12
A
21
A
22

1
=

A
1
11
(I + A
12
F
2
A
21
A
1
11
) A
1
11
A
12
F
2
F
2
A
21
A
1
11
F
2

where
F
2
= (A
22
A
21
A
1
11
A
12
)
1
.
The upper left block can also be written as
F
1
= (A
11
A
12
A
1
22
A
21
)
1
.
6 Linear system
Now lets consider the following system of linear equations:
a
11
x
1
+ a
12
x
2
+ a
1m
x
m
= b
1
a
21
x
1
+ a
22
x
2
+ a
2m
x
m
= b
2
.
.
.
a
n1
x
1
+ a
n2
x
2
+ a
nm
x
m
= b
n
We are interested in the properties of the solution to the above linear system which have n
equations and m unknowns. Using the notation of matrix, we can write it into:
Ax = b ()
where
A =

a
11
a
12
a
1m
a
21
a
22
a
2m
.
.
.
.
.
.
.
.
.
.
.
.
a
n1
a
n2
a
nm

, x =

x
1
x
2
.
.
.
x
m

, b =

b
1
b
2
.
.
.
b
n

9
A is called the coecient matrix of the system, and the matrix

A =

a
11
a
12
a
1m
b
1
a
21
a
22
a
2m
b
2
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
a
n1
a
n2
a
nm
b
n

is called augmented matrix of the system.

Denition 14 A Homogeneous (respectively, nonhomogeneous) Equation System is the one has
the form in () with b = 0 (respectively, b = 0)
6.1 A is nonsingular square matrix
In this special case, we can easily conclude that the equation system () has a unique solution,
which is given by:
x = A
1
b
When b = 0, the unique solution is 0 (zero vector). Remember the denition of linear depen-
dence, we can say that the columns of the matrix A are linear independent. This result gives the
following proposition:
Proposition 1 Let A be a square matrix, then the columns of A is linear independent i A is
nonsingular.
For this set of equation system, we have following theorem:
Theorem 5 Cremers Rule: let Ax = b be a linear equation system, suppose A is nonsingular,
then for all i = 1, 2, m, we have:
x
i
=
|B
i
|
|A|
where B
i
is the matrix obtained by replacing the i
th
column of A with b.
7 Eigenvalues and eigenvectors
7.1 Denitions
Let A be a n n matrix, be a real number. Consider the following equation system:
Ax = x ()
Notice that if c is the solution to the system (), then for any value of , c is also the solution
to the system. Hence sometimes, we normalize the candidates for x be such vectors that x x =1.
We can rewrite () as:
(AI) x = 0 ()
From the above section, we know we have non-zero solution i A I is singular, that is, i
|AI| = 0.
10
Denition 15 |AI| is called characteristic polynomial of A, |AI| = 0 is called character-
istic function of A. The solution to characteristic function is called eigenvalues (or characteristic
roots) of A. The solution vectors c associate with each eigenvalues are called eigenvectors (char-
acteristic vectors).
Remark 12 There always exists n eigenvalues for any matrices. Some of them may be the same,
and some (or all) of them may be complex numbers. If A is symmetric, however, all eigenvalues
are real numbers.
There are two facts about the characteristics function. Let
i
, i = 1 n, be the eigenvalues of
square matrix A, then:
1.

n
i=1

i
= tr (A).
2.

n
i=1

i
= |A|.
7.2 Diagonalization and Canonial Form (optional)
Let
i
, i = 1 n, be the eigenvalues of square matrices A, let c
i
be the associate eigenvectors.
The the equation system () can be written as
AC = C
where
C =

c
1
c
2
c
n

1
0 0
0
2
0
.
.
.
.
.
.
.
.
.
.
.
.
0 0
n

If C is invertible, then we have:

C
1
AC =
This brings out the following denition:
Denition 16 If there exist a matrices C such that C
1
AC = is a diagonal matrices, then A is
diagonalizable and is the canonial form of A.
Remark 13 The value of eigenvalues of a diagonalizable matrices are closely related to the stability
of corresponding (linear) dynamic system z
t+1
= Az
t
, which appears a lot in macroeconomics.
Then the question is under what condition, A matrix is diagonalizable. A immediate result is
given by the following theorems.
Theorem 6 A square matrix is diagonalizable i the eigenvectors of the matrix are linear inde-
pendent.
11
Theorem 7 For a square matrix A, if k eigenvalues
i
, i = 1 k are distinct, then the corre-
sponding eigenvalues are linear independent.
So the theorem says that a sucient condition for A to be diagonalizable is that it has n
distinct eigenvalues. However, in some cases the nn matrix A does not necessarily has n distinct
eigenvalues. The following two theorem are about those situations.
Theorem 8 Suppose
j
is a lfold root of the characteristic function of matrix A, then there are
at most l linear independent eigenvectors associate with
j
.
Theorem 9 Let
i
, i = 1 k be k distinct eigenvalues of nn matrix A, and let
i
is associated
with m(i) linear independent eigenvectors, denoted by c
i,l
i
, l
i
= 1, 2, m(i), then the set of
vectors {c
i,l
i
}
i=1,k;l
i
=1,m(i)
are linear independent.
Remark 14 Notice that

k
i=1
m(i) n.
7.3 Diagonalization of real symmetric matrix (optional)
A special class of matrices is real symmetric matrices which we will encounter a lot in applications.
It has special properties for its eigenvalues and eigenvectors.
Theorem 10 All eigenvalues of a real symmetric matrix are real.
Theorem 11 Let A be a n n real symmetric matrix, and let
i
, i = 1, , n be its eigenvalues,
and c
i
, i = 1, , n be corresponding eigenvectors. Then for all i = j, c

i
c
j
= 0. That is, all
eigenvectors of a real symmetric matrix are othogonal.
Since can write the system as:
AC = C
Remember we can normalize the eigenvectors be such vectors that c c = 1, then:
C

C = CC

1
c
1
c

1
c
2
c

1
c
n
c

2
c
1
c

2
c
2
c

2
c
n
.
.
.
.
.
.
.
.
.
.
.
.
c

n
c
1
c

n
c
2
c

n
c
n

= I
This implies that
C

= C
1
and
C

AC = C
1
AC =
Denition 17 A matrix Q is called othogonal, if all of its columns satisfy c

i
c
i
= 1 and for all
i = j, c

i
c
j
= 0.
Theorem 12 A real symmetric matrix is diagonalizable. Moreover, there exist othogonal matrix
Q such that Q
1
AQ = .
Corollary 1 Let
j
is a lfold root of the characteristic function of real symmetric matrix A, then
there are l linear independent eigenvectors associate with
j
.
12
8 Idempotent matrices
In section 1, we met symmetric idempotent matrices. Now we can explore more properties of this
kind of matrices.
Theorem 13 Let M
0
be a n n symmetric idempotent matrix, then:
1. Its eigenvalues are 1 or 0.
2. (M
0
) = tr (M
0
) = r
3. There exist othogonal matrix Q such that
Q
1
M
0
Q = =

I
rr
O
r(nr)
O
(nr)r
O
(nr)(nr)

Proof. Let be the eigenvalue of M

0
. Then we have from some c:
M
0
c =c
Premultiply both sides by M
0
:
M
2
0
c =M
0
c =
2
c
But M
2
0
= M
0
, hence

2
c =c
Since c = 0, we have:

2
=
So Its eigenvalues are 1 or 0.
As for 2, we have:
(M
0
) =

Q
1
M
0
Q

= ()
= tr ()
= tr

Q
1
M
0
Q

= tr (M
0
)
3 is obvious.
Remark 15 You will nd in econometrics class that when carrying out some hypothesis test, for
example, Chi-sqare test, the statistics are constructed on some symmetric idempotent matrices.
The numbers of the 0 in its eigenvalues, is equal to the degree of freedom of the test.
13
9 Quadratic form and denite matrices
Denition 18 A quadratic form is:
q = x

Ax =
n

i=1
n

j=1
x
i
x
j
a
ij
where A is a n n symmetric matrix.
Denition 19 An n n symmetric matrix A is called
1. positive (negative) denite if x

Ax >(<) 0 for all x = 0.

2. positive (negative) semidenite if x

Ax () 0 for all x = 0
Notice that:
x

Ax = x

QAQ
1
x
= x

QQ

x
= y

y
=
n

i=1

i
y
2
i
Hence we have the following theorem:
Theorem 14 Let A be an n n symmetric matrix, let
i
be its eigenvalues. Then:
1. positive (negative) denite if
i
> (<) 0 for all i = 1, , n.
2. positive (negative) semidenite if
i
> (<) 0 for i = 1, , k and
i
= 0 for i = k + 1, , n,
where 1 k n 1.
Denition 20 Le A be n m matrix, then its k
th
order leading principal submatrix, denoted by
A
k
, is obtained by deleting the last n k columns and rows. Let A
P
be an permutation of A, and
let A
P
k
be the leading principal submatrix of A
P
.
Theorem 15 Let A be an n n matrix,
1. and if A is symmetric, then A is positive denite i |A
k
| > 0 for all k = 1, , n.
2. and if A is symmetric, then A is negative denite i (1)
k
|A
k
| > 0.
3. then A is positive semidenite i

A
P
k

0 for all k = 1, , n and all permutation.

4. then A is negative semidenite i (1)
k

A
P
k

0 for all k = 1, , n and all permutation.

Here are some other properties:
14
If A is positive semidenite denite, then |A| 0. But the reverse is not true.
If A is positive denite, then so is A
1
.
If A is a n m matrix with full column rank and n > m, then A

A is positive denite and AA

is positive semidenite.
If A is positive denite and B is nonsingular, then B

AB is positive denite.
Matrix A is positive denite i the symmetric matrix A + A

is positive denite.
10 Matrix Derivatives
Let a and x be n 1 vectors. Let f(x) = a

x. Then
f(x)
x
= a

.
That is, take a derivative of a scalarvalued function with respect to a column vector results in a
row vector. For a n n matrix A, let g(x) = x

Ax, then
g(x)
x
= 2x

A.
11 Kronecker Product (optional)
Denition 21 Let A be an nm matrix, and B be a l k matrix, then the kronecker product
of A and B is dened by:
AB =

a
11
B a
12
B a
1m
B
a
21
B a
22
B a
2m
B
.
.
.
.
.
.
.
.
.
.
.
.
a
n1
B a
n2
B a
nm
B

nlmk
For Kronecker Product, we have following properties:
If A is n n, B is l l, then |AB| = |A|
n
|B|
l
; (AB)

= A

; tr (AB) = tr (A) tr (B)

If A and B are invertible, then (AB)
1
=

A
1
B
1

A(B C) = (AB) C
A(B + C) = AB + AC
aAbB = abAB
(AB) (C D) = AC BD
(AB) = (A) (B)
15