CSCE 633: Machine Learning

Instructor: Dr. Zhangyang (Atlas) Wang

Email: atlaswang@tamu.edu
Website: www.atlaswang.com
Lecture 2: Linear Algebra and Matrix Analysis
Vector Norm

Discrete Vector Continuous Functional

(Finite-dimensional case) (Infinite-dimensional case)
Vector Norm

Quick Test: Are They Norms?

Vector Norm

Remarks:
• Finite-dimensional real or complex vector space: all norms are equivalent. How to Prove? (Link)
• Infinite-dimensional vector spaces: not all norms are equivalent.
• Why norm equivalence is important: defining the same notions of continuity and convergence
• The uniform structure defined by equivalent norms on the vector space is called “uniformly
isomorphic”
Vector Angle
Vector Angle

The Importance of Choosing the Right “Distance” to Measure (a.k.a. Metric)

Linear Independence
Matrix Rank
Range and Null Space
Range and Null Space

Rank–Nullity Theorem (plain language):

The rank and the nullity of a matrix add up to the number of its columns.

Proof: By properties of linear transformations https://en.wikipedia.org/wiki/Rank–nullity_theorem

Eigenvalues and eigenvectors
Eigenface
• The eigenvectors of the covariance
matrix associated with a large set
of normalized pictures of faces are
called Eigenfaces.
• Eigenfaces reflect a set of
“representative appearances” in
the given collection of faces
• This is an example of principal
component analysis (PCA) too.
Matrix norms
Matrix norms
Condition number

“Ill-Posed” when solving the

inverse of a linear system
Orthogonality
Singular Value Decomposition (SVD)
• SVD is extremely powerful and useful. Numerous practical applications exploit key SVD properties,
i.e., its relation to the rank of a matrix and its optimal low-rank approximation of a given matrix.
Singular Value Decomposition (SVD)
Singular Value Decomposition (SVD)

Extended Reading: Numerous matrix factorization models: LU, QR, Cholesky, Nonnegative Matrix
Factorization (NMF)… https://en.wikipedia.org/wiki/Matrix_decomposition
SVD: Eckart–Young–Mirsky Theorem
Let’s do a proof!
SVD: Eckart–Young–Mirsky Theorem
SVD: How to Compute
• The most commonly used SVD algorithm is found in Matlab and in the LAPACK linear algebra library
(http://www.netlib.org/lapack/)

Householder transformations.

QR Decomposition (variant)
Checklist: What to take home?
• Vector norm; vector angle; matrix norm
• Subspace; basis; linear independence and orthogonality
• Rank; range and null space; condition number
• EVD and SVD; E-Y-M Theorem

Next Class
• Vector Space and Optimization