EE5121: Optimization Methods

HW #1: Norms

Exercise 1. Find the sup and the inf of the following sets:

1. A1 = { n1 ; n = 1, 2, 3, . . .}

2. A2 = {21/n , n = 1, 2, . . .}

3. A3 = {x : arcsin(x) < 1/2}

4. A4 = (3, 4)

5. A5 = [3, 4]

6. A6 = (3, 4) [6, 10]

7. A7 = (3, 4) (2, 6)

8. A8 = (3, 4) [6, 10]

x
9. A9 = { 1+x : x R, x 0}

10. A10 = {x : x [1, 1]}

11. A11 = {x + y : (x, y) [1, 1]2 }

12. A12 = {x + y + z : (x, y, z) [1, 1]3 }

For the sets A10 , A11 , A12 , observe where the infimum and the supremum occur. Later in the
course, we will examine the suprema (infima) of such sets in more detail.

Exercise 2. 1. Given A R, can sup(A) = inf(A)? If so, characterize all such A.

2. Intuitively, for what class of subsets of R is sup(A) = max(A) true?

function kxkp : Rn [0, ), p 1

Exercise 3. In this problem, we will establish that the P
satisfies the properties of a norm (recall that kxkp = ( ni=1 |xi |p )1/p ). The difficulty lies in
proving the triangle inequality. To prove the triangle inequality, the following inequality is
generally used:
(Holders inequality) For any vectors a, b Rn ,

aT b kakp kbkq , p1 + q 1 = 1.

When p1 + q 1 = 1, p and q are called the conjugate exponents of each other. Observe that
you get the famous Cauchy-Schwartz inequality by setting p = q = 2. Note that the combination
p = 1, q = is perfectly valid.

#1: Norms-1
Proof. We now start with the proof of triangle inequality. This particular triangular inequality
of p-th norm is also called as Minkowski inequality.

n
(a) X
kx + ykpp = |xi + yi |p
i=1
n
(b)
X
(|xi | + |yi |)|xi + yi |p1
i=1
n
X n
X
p1
= |xi ||xi + yi | + |yi ||xi + yi |p1
i=1 i=1
n
!1 1 n
!1 1
(c) p p
p p
(p1) p1 (p1) p1
X X
kxkp |xi + yi | + kykp |xi + yi |
i=1 i=1

Justify the steps (a), (b), (c) clearly and to the point, and complete the remaining proof
of kx + ykp kxkp + kykp .
Exercise 4. Show that as p , k kp k k .
Exercise 5. 1. Plot B(o, 1) when the norm is given by k.k1 in R2 .
2. Plot B(o, 1) when the norm is given by k.k2 in R2 .
3. Plot B(o, 1) when the norm is given by k.k in R2 .
4. Plot B(o, 1) when the norm is given by k.k3 in R2 . (approximately)
Here, o represents the origin in the appropriate space. Explain what equivalence of these
norms is, using these plots.
Exercise 6. A norm that is commonly used in information theory and computer science is
the Hamming norm and is defined as the Hamming distance to the zero vector. For any x Rn ,
the hamming norm is defined as
kxkh = Number of non-zero coordinates of x.
1. Plot B(o, 1), B(o, 1.5), B(o, 2), B(o, 2.5), B(o, 4) in R2 with the Hamming norm (remember
the strict inequality in the definition of the Ball)
2. Is kxkh is a norm? What if the norm is defined on the space {0, 1}n , rather than the real
space Rn .
Exercise 7. Let A be a m n real matrix. For any x Rn , define
kxkA = kAxk2 .
Is kxkA a norm in Rm for any matrix A?. What properties of A will make k.kA a norm? (Hint:
Check when the norm becomes zero, and relate it to the Null space of the matrix. If you have
not heard of null space, it will be a good time to review basic linear algebra, especially solutions
of linear equations.)

#1: Norms-2
 2 p
Exercise 8. Let A be any n n real matrix. Define f : Rn R+ as f (A) = tr(AT A). Is f
a valid norm on the space of all n n real matrices? If not, under what condition will f be a
valid norm?
Exercise 9. Let A be a n n real matrix. For any x Rn , define

kxkA = xT Ax.
Is kxkA a norm in Rn a norm for any matrix A? (prove or disprove) What are the properties
of A that will make k.kA a norm? Hint: positive definite matrices.
Exercise 10 (Induced Matrix operator norms). The norms on matrices defined below are also
called as induced norms since we use norms on Rn (the domain) and Rm (the image) to define
them. For A, a m n real matrix, define
 
kAxk
kAk = max , x 6= 0 . (1)
kxk
An alternative definition is
kAk = max {kAxk, kxk = 1} . (2)
From the second definition, we observe that the norm measures the maximum norm of a point
in the image of the Ball B(o, 1) after the linear transformation using the matrix A.
1. Check that the induced matrix norm satisfies the properties of norms.
kAxkp
2. Let kAkp be the induced norm maxx6=0 kxkp (or alternatively use the other definition).
Show that
n
X
kAk1 = max |aij |.
1jn
i=1
Essentially, kAk1 equals the maximum of absolute column sums.
3. Show that
m
X
kAk = max |aij |.
1im
j=1

Essentially, kAk equals the maximum of absolute row sums.

4. The kAk2 norm possess a beautiful property, which we will now prove:
q
kAk2 = max (AT A) = max (A).

Here, max is the maximum eigenvalue of AT A and max (A) is the maximum singular
value of A.

Proof.
kAk22 = max {kAxk22 }
kxk=1
(a)
= max {(Ax)T Ax} = max {(xT AT Ax}
kxk=1 kxk=1

#1: Norms-3
 Let U T U be the eigenvalue decomposition of the matrix AT A (Existence and proper-
ties?). Then

kAk22 = max {(xT AT Ax}

kxk=1
(b)
= max {(xU )T U x} = max {xT x}
kxk=1 kxk=1
(c)
= max

Justify the steps (a), (b) and (c), and fill in the missing steps in a clear manner.

5. Prove that kAxk kAkkxk, where kAk is the operator norm. Such matrix norms are
called consistent norms. (Hint: Use the definition (1)).

6. Let A be a n n symmetric matrix. Then, show that the spectral radius of A denoted by
(A) = max1in (|i |) satisfies the following inequality for any consistent matrix norm:

(A) kAk.

(Hint: Try to use an eigenvector of A and consistency property).

7. Multiplicative property: If A and B are n n matrices, for any matrix norm k.k prove
that
kABk kAkkBk.
( Hint: Use kAxk kAkkxk)

8. Now prove that (A) kAk k1/k , for any k.

The spectral radius can also be defined as (A) = limk kAk k1/k . (You dont have to
prove this).

9. Prove that the norms obtained from (1) and (2) are equal.
Exercise 11. (Hierarchy of spaces)
1. Let (X, k k) be a normed linear space over R. For x, y X, show that d(x, y) , kx yk
is a valid metric.

2. Let X be a vector space over R. Consider a function < , >: X X R that satisfies
the following axioms:

< x, x > 0 and < x, x >= 0 = x = 0, x X.

< x + y, z > = < x, z > + < y, z >, x, y, z X and , R.
< x, y >=< y, x >, x, y X.

Such a function is known as an inner product on X, and (X, < , >) is called an inner

product space. Show that f : X X R+ defined as f (x) = < x, x > is a valid norm
on X.

#1: Norms-4
This exercise serves to show that every inner product space is a normed linear space, and that
every normed linear space is a metric space. With appropriate counter-examples, it can be
shown that not every metric space is a normed linear space (in fact, one can come up metric
spaces that are not even vector spaces, let alone normed linear spaces!), and that not every
normed linear space is an inner product space.

Exercise 12. Prove the following for the setting case of finite-dimensional Euclidean spaces.

1. k k2 = k k2 .

2. k k1 = k k .

3. k k = k k1 .

Exercise 13. Generalize the above question to prove that p-th norm is the dual of q-the norm,
where p and q are conjugate exponents of each other, for a generic 1 p .

#1: Norms-5