Linear Algebra and Applications:

Numerical Linear Algebra

David S. Watkins
watkins@math.wsu.edu

Department of Mathematics
Washington State University

IMA Summer Program, 2008 p.

My Pledge to You

IMA Summer Program, 2008 p.

My Pledge to You
I promise not to cover as much material
as I previously claimed I would.

IMA Summer Program, 2008 p.

Resources

IMA Summer Program, 2008 p.

Resources (a biased list)

IMA Summer Program, 2008 p.

Resources (a biased list)
David S. Watkins, Fundamentals of Matrix
Computations, Second Edition, John Wiley and
Sons, 2002. (FMC)

IMA Summer Program, 2008 p.

Resources (a biased list)
David S. Watkins, Fundamentals of Matrix
Computations, Second Edition, John Wiley and
Sons, 2002. (FMC)
David S. Watkins, The QR algorithm revisited,
SIAM Review, 50 (2008), pp. 133145.

IMA Summer Program, 2008 p.

Resources (a biased list)
David S. Watkins, Fundamentals of Matrix
Computations, Second Edition, John Wiley and
Sons, 2002. (FMC)
David S. Watkins, The QR algorithm revisited,
SIAM Review, 50 (2008), pp. 133145.
David S. Watkins, The Matrix Eigenvalue Problem,
GR and Krylov Subspace Methods, SIAM, 2007.

IMA Summer Program, 2008 p.

Leslie Hogben, Handbook of Linear Algebra,
Chapman and Hall/CRC, 2007.

IMA Summer Program, 2008 p.

Leslie Hogben, Handbook of Linear Algebra,
Chapman and Hall/CRC, 2007.
Lloyd N. Trefethen and David Bau, III, Numerical
Linear Algebra, SIAM, 1997.

IMA Summer Program, 2008 p.

Leslie Hogben, Handbook of Linear Algebra,
Chapman and Hall/CRC, 2007.
Lloyd N. Trefethen and David Bau, III, Numerical
Linear Algebra, SIAM, 1997.
James W. Demmel, Applied Numerical Linear
Algebra, SIAM, 1997.

IMA Summer Program, 2008 p.

Leslie Hogben, Handbook of Linear Algebra,
Chapman and Hall/CRC, 2007.
Lloyd N. Trefethen and David Bau, III, Numerical
Linear Algebra, SIAM, 1997.
James W. Demmel, Applied Numerical Linear
Algebra, SIAM, 1997.
G. H. Golub and C. F. Van Loan, Matrix
Computations, Third Edition, Johns Hopkins
University Press, 1996.

IMA Summer Program, 2008 p.

Common Linear Algebra
Computations

IMA Summer Program, 2008 p.

Common Linear Algebra
Computations
linear system Ax = b

IMA Summer Program, 2008 p.

Common Linear Algebra
Computations
linear system Ax = b
overdetermined linear system Ax = b

IMA Summer Program, 2008 p.

Common Linear Algebra
Computations
linear system Ax = b
overdetermined linear system Ax = b
eigenvalue problem Av = v

IMA Summer Program, 2008 p.

Common Linear Algebra
Computations
linear system Ax = b
overdetermined linear system Ax = b
eigenvalue problem Av = v
various generalized eigenvalue problems,
e.g. Av = Bv

IMA Summer Program, 2008 p.

Linear Systems

IMA Summer Program, 2008 p.

Linear Systems
Ax = b, n n, nonsingular, real or complex

IMA Summer Program, 2008 p.

Linear Systems
Ax = b, n n, nonsingular, real or complex
Examples: FMC 1.2, 7.1; any linear algebra text

IMA Summer Program, 2008 p.

Linear Systems
Ax = b, n n, nonsingular, real or complex
Examples: FMC 1.2, 7.1; any linear algebra text
Major tools:
Gaussian elimination (LU Decomp.)
various iterative methods

IMA Summer Program, 2008 p.

Overdetermined Linear Systems

IMA Summer Program, 2008 p.

Overdetermined Linear Systems
Ax = b, n m, n > m

IMA Summer Program, 2008 p.

Overdetermined Linear Systems
Ax = b, n m, n > m
often n m

IMA Summer Program, 2008 p.

Overdetermined Linear Systems
Ax = b, n m, n > m
often n m
Example: fitting data by a straight line

IMA Summer Program, 2008 p.

Overdetermined Linear Systems
Ax = b, n m, n > m
often n m
Example: fitting data by a straight line
minimize k b Ax k2 (least squares)

IMA Summer Program, 2008 p.

Overdetermined Linear Systems
Ax = b, n m, n > m
often n m
Example: fitting data by a straight line
minimize k b Ax k2 (least squares)
Major tools:
QR decomposition
singular value decomposition

IMA Summer Program, 2008 p.

Eigenvalue Problems

IMA Summer Program, 2008 p.

Eigenvalue Problems
standard: Av = v, n n, real or complex

IMA Summer Program, 2008 p.

Eigenvalue Problems
standard: Av = v, n n, real or complex
Examples: FMC 5.1

IMA Summer Program, 2008 p.

Eigenvalue Problems
standard: Av = v, n n, real or complex
Examples: FMC 5.1
generalized: Av = Bv

IMA Summer Program, 2008 p.

Eigenvalue Problems
standard: Av = v, n n, real or complex
Examples: FMC 5.1
generalized: Av = Bv
Examples: FMC 6.7

IMA Summer Program, 2008 p.

Eigenvalue Problems
standard: Av = v, n n, real or complex
Examples: FMC 5.1
generalized: Av = Bv
Examples: FMC 6.7
product: A1 A2

IMA Summer Program, 2008 p.

Eigenvalue Problems
standard: Av = v, n n, real or complex
Examples: FMC 5.1
generalized: Av = Bv
Examples: FMC 6.7
product: A1 A2
Examples: generalized (AB 1 ), SVD (A A)

IMA Summer Program, 2008 p.

Eigenvalue Problems
standard: Av = v, n n, real or complex
Examples: FMC 5.1
generalized: Av = Bv
Examples: FMC 6.7
product: A1 A2
Examples: generalized (AB 1 ), SVD (A A)
quadratic: (2 K + G + M )v = 0

IMA Summer Program, 2008 p.

Sizes of Linear Algebra Problems

IMA Summer Program, 2008 p.

Sizes of Linear Algebra Problems
small

IMA Summer Program, 2008 p.

Sizes of Linear Algebra Problems
small
medium

IMA Summer Program, 2008 p.

Sizes of Linear Algebra Problems
small
medium
large

IMA Summer Program, 2008 p.

Solving Linear Systems:

IMA Summer Program, 2008 p. 1

Solving Linear Systems:
small problems

IMA Summer Program, 2008 p. 1

Solving Linear Systems:
small problems
Ax = b, n n, n small

IMA Summer Program, 2008 p. 1

Solving Linear Systems:
small problems
Ax = b, n n, n small
store A conventionally

IMA Summer Program, 2008 p. 1

Solving Linear Systems:
small problems
Ax = b, n n, n small
store A conventionally
solve using Gaussian elimination

IMA Summer Program, 2008 p. 1

Solving Linear Systems:
small problems
Ax = b, n n, n small
store A conventionally
solve using Gaussian elimination
A = LU

IMA Summer Program, 2008 p. 1

Solving Linear Systems:
small problems
Ax = b, n n, n small
store A conventionally
solve using Gaussian elimination
A = LU
P A = LU (partial pivoting)

IMA Summer Program, 2008 p. 1

Solving Linear Systems:
small problems
Ax = b, n n, n small
store A conventionally
solve using Gaussian elimination
A = LU
P A = LU (partial pivoting)
forward and back substitution

IMA Summer Program, 2008 p. 1

Solving Linear Systems:
small problems
Ax = b, n n, n small
store A conventionally
solve using Gaussian elimination
A = LU
P A = LU (partial pivoting)
forward and back substitution
Questions: cost?,

IMA Summer Program, 2008 p. 1

Solving Linear Systems:
small problems
Ax = b, n n, n small
store A conventionally
solve using Gaussian elimination
A = LU
P A = LU (partial pivoting)
forward and back substitution
Questions: cost?, accuracy? (FMC Ch. 2)

IMA Summer Program, 2008 p. 1

Positive Definite Case

IMA Summer Program, 2008 p. 1

Positive Definite Case
A = A , x Ax > 0 for all x 6= 0

IMA Summer Program, 2008 p. 1

Positive Definite Case
A = A , x Ax > 0 for all x 6= 0
A = R R Cholesky decomposition

IMA Summer Program, 2008 p. 1

Positive Definite Case
A = A , x Ax > 0 for all x 6= 0
A = R R Cholesky decomposition
symmetric variant of Gaussian elimination

IMA Summer Program, 2008 p. 1

Positive Definite Case
A = A , x Ax > 0 for all x 6= 0
A = R R Cholesky decomposition
symmetric variant of Gaussian elimination
flop count is halved

IMA Summer Program, 2008 p. 1

Solving Linear Systems:

IMA Summer Program, 2008 p. 1

Solving Linear Systems:
medium problems

IMA Summer Program, 2008 p. 1

Solving Linear Systems:
medium problems
Larger problems are usually sparser.

IMA Summer Program, 2008 p. 1

Solving Linear Systems:
medium problems
Larger problems are usually sparser.
Use sparse data structure.

IMA Summer Program, 2008 p. 1

Solving Linear Systems:
medium problems
Larger problems are usually sparser.
Use sparse data structure.
sparse Gaussian elimination

IMA Summer Program, 2008 p. 1

Solving Linear Systems:
medium problems
Larger problems are usually sparser.
Use sparse data structure.
sparse Gaussian elimination
A = LU

IMA Summer Program, 2008 p. 1

Solving Linear Systems:
medium problems
Larger problems are usually sparser.
Use sparse data structure.
sparse Gaussian elimination
A = LU
factors usually less sparse than A,

IMA Summer Program, 2008 p. 1

Solving Linear Systems:
medium problems
Larger problems are usually sparser.
Use sparse data structure.
sparse Gaussian elimination
A = LU
factors usually less sparse than A, but still sparse

IMA Summer Program, 2008 p. 1

Solving Linear Systems:
medium problems
Larger problems are usually sparser.
Use sparse data structure.
sparse Gaussian elimination
A = LU
factors usually less sparse than A, but still sparse
Crucial question: Can factors fit in main memory?

IMA Summer Program, 2008 p. 1

Solving Linear Systems:

IMA Summer Program, 2008 p. 1

Solving Linear Systems:
large problems

IMA Summer Program, 2008 p. 1

Solving Linear Systems:
large problems
L and U factors may be too large to store . . .

IMA Summer Program, 2008 p. 1

Solving Linear Systems:
large problems
L and U factors may be too large to store . . .
Use an iterative method.

IMA Summer Program, 2008 p. 1

Solving Linear Systems:
large problems
L and U factors may be too large to store . . .
Use an iterative method.
direct vs. iterative methods

IMA Summer Program, 2008 p. 1

Solving Linear Systems:
large problems
L and U factors may be too large to store . . .
Use an iterative method.
direct vs. iterative methods
Some buzz words: descent method, conjugate
gradients (CG), GMRES, . . .

IMA Summer Program, 2008 p. 1

Solving Linear Systems:
large problems
L and U factors may be too large to store . . .
Use an iterative method.
direct vs. iterative methods
Some buzz words: descent method, conjugate
gradients (CG), GMRES, . . .
preconditioners,

IMA Summer Program, 2008 p. 1

Solving Linear Systems:
large problems
L and U factors may be too large to store . . .
Use an iterative method.
direct vs. iterative methods
Some buzz words: descent method, conjugate
gradients (CG), GMRES, . . .
preconditioners, and on and on.

IMA Summer Program, 2008 p. 1

Solving Linear Systems:
large problems
L and U factors may be too large to store . . .
Use an iterative method.
direct vs. iterative methods
Some buzz words: descent method, conjugate
gradients (CG), GMRES, . . .
preconditioners, and on and on.
FMC Chapter 7

IMA Summer Program, 2008 p. 1

Solving Linear Systems:
large problems
L and U factors may be too large to store . . .
Use an iterative method.
direct vs. iterative methods
Some buzz words: descent method, conjugate
gradients (CG), GMRES, . . .
preconditioners, and on and on.
FMC Chapter 7
Richard Barrett et. al., Templates for the Solution of
Linear Systems, . . . , SIAM 1994. (FREE!!!)
IMA Summer Program, 2008 p. 1
Moving On

IMA Summer Program, 2008 p. 1

Moving On
Orthogonal Transformations

IMA Summer Program, 2008 p. 1

Moving On
Orthogonal Transformations
generally useful computing tools

IMA Summer Program, 2008 p. 1

Moving On
Orthogonal Transformations
generally useful computing tools
sticking to real case for simplicity

IMA Summer Program, 2008 p. 1

Moving On
Orthogonal Transformations
generally useful computing tools
sticking to real case for simplicity
Pn
standard inner product: hx, yi = j=1 xj yj

IMA Summer Program, 2008 p. 1

Moving On
Orthogonal Transformations
generally useful computing tools
sticking to real case for simplicity
Pn
standard inner product: hx, yi = j=1 xj yj
P 1/2
n 2
Euclidean norm: k x k2 = x
j=1 j

IMA Summer Program, 2008 p. 1

Moving On
Orthogonal Transformations
generally useful computing tools
sticking to real case for simplicity
Pn
standard inner product: hx, yi = j=1 xj yj
P 1/2
n 2
Euclidean norm: k x k2 = x
j=1 j
p
k x k2 = hx, xi

IMA Summer Program, 2008 p. 1

Moving On
Orthogonal Transformations
generally useful computing tools
sticking to real case for simplicity
Pn
standard inner product: hx, yi = j=1 xj yj
P 1/2
n 2
Euclidean norm: k x k2 = x
j=1 j
p
k x k2 = hx, xi
definition of orthogonal: QT = Q1

IMA Summer Program, 2008 p. 1

Moving On
Orthogonal Transformations
generally useful computing tools
sticking to real case for simplicity
Pn
standard inner product: hx, yi = j=1 xj yj
P 1/2
n 2
Euclidean norm: k x k2 = x
j=1 j
p
k x k2 = hx, xi
definition of orthogonal: QT = Q1
properties of orthogonal matrices
IMA Summer Program, 2008 p. 1
Elementary Reflectors

IMA Summer Program, 2008 p. 1

Elementary Reflectors
= Householder transformations

IMA Summer Program, 2008 p. 1

Elementary Reflectors
= Householder transformations
one of two major classes of computationally useful
orthogonal transformations

IMA Summer Program, 2008 p. 1

Elementary Reflectors
= Householder transformations
one of two major classes of computationally useful
orthogonal transformations
Q = I 2uuT , k u k2 = 1

IMA Summer Program, 2008 p. 1

Elementary Reflectors
= Householder transformations
one of two major classes of computationally useful
orthogonal transformations
Q = I 2uuT , k u k2 = 1
geometric action

IMA Summer Program, 2008 p. 1

Elementary Reflectors
= Householder transformations
one of two major classes of computationally useful
orthogonal transformations
Q = I 2uuT , k u k2 = 1
geometric action
Qx = y

IMA Summer Program, 2008 p. 1

Elementary Reflectors
= Householder transformations
one of two major classes of computationally useful
orthogonal transformations
Q = I 2uuT , k u k2 = 1
geometric action
Qx = y
creating zeros

IMA Summer Program, 2008 p. 1

Elementary Reflectors
= Householder transformations
one of two major classes of computationally useful
orthogonal transformations
Q = I 2uuT , k u k2 = 1
geometric action
Qx = y
creating zeros
details: FMC Chapter 3

IMA Summer Program, 2008 p. 1

Elementary Reflectors
= Householder transformations
one of two major classes of computationally useful
orthogonal transformations
Q = I 2uuT , k u k2 = 1
geometric action
Qx = y
creating zeros
details: FMC Chapter 3
QR decomposition
IMA Summer Program, 2008 p. 1
Uses of the QR Decomposition

IMA Summer Program, 2008 p. 1

Uses of the QR Decomposition
Ax = b, nn

IMA Summer Program, 2008 p. 1

Uses of the QR Decomposition
Ax = b, n n
overdetermined system

IMA Summer Program, 2008 p. 1

Uses of the QR Decomposition
Ax = b, n n
overdetermined system
orthonormalizing vectors

IMA Summer Program, 2008 p. 1

The Gram-Schmidt Process

IMA Summer Program, 2008 p. 1

The Gram-Schmidt Process
orthonormalization of vectors

IMA Summer Program, 2008 p. 1

The Gram-Schmidt Process
orthonormalization of vectors
relationship to QR decomposition

IMA Summer Program, 2008 p. 1

The Gram-Schmidt Process
orthonormalization of vectors
relationship to QR decomposition
reorthogonalization

IMA Summer Program, 2008 p. 1

The SVD

IMA Summer Program, 2008 p. 1

The SVD
singular value decomposition

IMA Summer Program, 2008 p. 1

The SVD
singular value decomposition
A = U V T

IMA Summer Program, 2008 p. 1

The SVD
singular value decomposition
A = U V T
product eigenvalue problem

IMA Summer Program, 2008 p. 1

The SVD
singular value decomposition
A = U V T
product eigenvalue problem
FMC Chapter 4

IMA Summer Program, 2008 p. 1

The SVD
singular value decomposition
A = U V T
product eigenvalue problem
FMC Chapter 4
numerical rank determination

IMA Summer Program, 2008 p. 1

The SVD
singular value decomposition
A = U V T
product eigenvalue problem
FMC Chapter 4
numerical rank determination
solution of least-squares problem

IMA Summer Program, 2008 p. 1

End of Part I

IMA Summer Program, 2008 p. 1