You are on page 1of 184

Notes for ENEE 664: Optimal Control

Andre L. Tits
DRAFT
July 2011
Contents
1 Motivation and Scope 3
1.1 Some Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Scope of the Course . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2 Linear Optimal Control: Some (Reasonably) Simple Cases 11
2.1 Free terminal state, unconstrained, quadratic cost . . . . . . . . . . . . . . . 12
2.1.1 Finite horizon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.1.2 Innite horizon, LTI systems . . . . . . . . . . . . . . . . . . . . . . . 18
2.2 Free terminal state, constrained control values, linear cost . . . . . . . . . . . 24
2.3 Fixed terminal state, unconstrained, quadratic cost . . . . . . . . . . . . . . 27
2.4 More general optimal control problems . . . . . . . . . . . . . . . . . . . . . 35
3 Unconstrained Optimization 37
3.1 First order condition of optimality . . . . . . . . . . . . . . . . . . . . . . . . 37
3.2 Steepest descent method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.3 Introduction to convergence analysis . . . . . . . . . . . . . . . . . . . . . . 40
3.4 Minimization of convex functions . . . . . . . . . . . . . . . . . . . . . . . . 48
3.5 Second order optimality conditions . . . . . . . . . . . . . . . . . . . . . . . 49
3.6 Conjugate direction methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.7 Rates of convergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.8 Newtons method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
3.9 Variable metric methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4 Constrained Optimization 67
4.1 Abstract Constraint Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
4.2 Equality Constraints - First Order Conditions . . . . . . . . . . . . . . . . . 70
4.3 Equality Constraints Second Order Conditions . . . . . . . . . . . . . . . . 77
4.4 Inequality Constraints First Order Conditions . . . . . . . . . . . . . . . . 80
4.5 Mixed Constraints First Order Conditions . . . . . . . . . . . . . . . . . . 90
4.6 Mixed Constraints Second order Conditions . . . . . . . . . . . . . . . . . 92
4.7 Glance at Numerical Methods for Constrained Problems . . . . . . . . . . . 94
4.8 Sensitivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
4.9 Duality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
4.10 Linear programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
Copyright c _19932011, Andre L. Tits. All Rights Reserved 1
CONTENTS
5 Calculus of Variations and Optimal Control 115
5.1 Introduction to the calculus of variations . . . . . . . . . . . . . . . . . . . . 115
5.2 Discrete-Time Optimal Control . . . . . . . . . . . . . . . . . . . . . . . . . 120
5.3 Continuous-Time Optimal Control - Linear Systems . . . . . . . . . . . . . . 123
5.4 Continuous-Time Optimal Control; Nonlinear Systems . . . . . . . . . . . . 127
5.5 Dynamic programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
A Generalities on Vector Spaces 147
B On Dierentiability and Convexity 163
B.1 Dierentiability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
B.2 Some elements of convex analysis . . . . . . . . . . . . . . . . . . . . . . . . 171
B.3 Acknowledgment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
2 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
Chapter 1
Motivation and Scope
1.1 Some Examples
We give some examples of design problems in engineering that can be formulated as math-
ematical optimization problems. Although we emphasize here engineering design, optimiza-
tion is widely used in other elds such as economics or operations research. Such examples
can be found, e.g., in [18].
Example 1.1 Design of an operational amplier (opamp)
Suppose the following features (specications) are desired
1. a large gain-bandwidth product
2. sucient stability
3. low power dissipation
In this course, we deal with parametric optimization. This means, for this example, that
we assume the topology of the circuit has already been chosen, the only freedom left being
the choice of the value of a number of design parameters (resistors, capacitors, various
transistor parameters). In real world, once the parametric optimization has been performed,
the designer will possibly decide to modify the topology of his circuit, hoping to be able to
achieve better performances. Another parametric optimization is then performed. This loop
may be repeated many times.
To formulate the opamp design problem as an optimization problem, one has to spec-
ify one (possibly several) objective function(s) and various constraints. We decide for the
following goal:
minimize the power dissipated
subject to gain-bandwidth product M
1
(given)
frequency response M
2
at all frequencies.
The last constraint will prevent two high a peaking in the frequency response, thereby
ensuring sucient closed-loop stability margin. We now denote by x the vector of design
parameters
Copyright c _19932011, Andre L. Tits. All Rights Reserved 3
Motivation and Scope
x = (R
1
, R
2
, . . . , C
1
, C
2
, . . . ,
i
, . . .) R
n
For any given x, the circuit is now entirely specied and the various quantities mentioned
above can be computed. More precisely, we can dene
P(x) = power dissipated
GB(x) = gain-bandwidth product
FR(x, ) = frequency response, as a function of the frequency .
We then write the optimization problem as
minP(x)[GB(x) M
1
, FR(x, ) M
2
(1.1)
where = [
1
,
2
] is a range of critical frequencies. To obtain a canonical form, we now
dene
f(x) := P(x) (1.2)
g(x) := M
1
GB(x) (1.3)
(x, ) := FR(x, ) M
2
(1.4)
and we obtain
minf(x)[g(x) 0, (x, ) 0 (1.5)
Note. We will systematically use notations such as
minf(x)[g(x) 0
not to just indicate the minimum value, but rather as a short-hand for
minimize f(x)
subject to g(x) 0
More generally, one would have
minf(x)[g
i
(x) 0, i = 1, 2, . . . , m,
i
(x, ) 0,
i
, i = 1, . . . , k. (1.6)
If we dene g : R
n
R
m
by
g(x) =
_

_
g
1
(x)
.
.
.
g
m
(x)
_

_
(1.7)
4 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
1.1 Some Examples
and, assuming that all the
i
s are identical, if we dene : R
n
R
k
by
(x, ) =
_

1
(x, )
.
.
.

k
(x, )
_

_
(1.8)
we obtain again
minf(x)[g(x) 0, (x, ) 0 (1.9)
[This is called a semi-innite optimization problem: nitely many variables, innitely many
constraints.]
Note. If we dene

i
(x) = sup

i
(x, ) (1.10)
(1.9) is equivalent to
minf(x)[g(x) 0, (x) 0 (1.11)
(more precisely, x[(x, ) 0 = x[(x) 0) and (x) can be absorbed into
g(x).
Exercise 1.1 Prove the equivalence between (1.9) and (1.11). (To prove A = B, prove
A B and B A.)
This transformation may not be advisable, for the following reasons:
(i) some potentially useful information (e.g., what are the critical values of ) is lost
when replacing (1.9) by (1.11)
(ii) for given x, (x) may not be computable exactly in nite time (this computation
involves another optimization problem)
(iii) may not be smooth even when is, as shown in the exercise below. Thus (1.11) may
not be solvable by classical methods.
Exercise 1.2 Suppose that : R
n
R is continuous and that is compact, so that
the sup in (1.10) can be written as a max.
(a) Show that is continuous.
(b) By exhibiting a counterexample, show that there might not exist a continuous function
() such that, for all x, (x) = (x, (x)).
Exercise 1.3 Again referring to (1.10), show, by exhibiting counterexamples, that continuity
of is no longer guaranteed if either (i) is compact but is merely continuous in each
variable separately or (ii) is jointly continuous but is not compact, even when the sup
in (1.10) is achieved for all x.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 5
Motivation and Scope
Exercise 1.4 Referring still to(1.10), exhibit an example where C

(all derivatives
exist and are continuous), where is compact, but where is not everywhere dierentiable.
However, in this course, we will limit ourselves mostly to classical (non semi-innite) prob-
lems (and will generally assume continuous dierentiability), i.e., to problems of the form
minf(x) : g(x) 0, h(x) = 0
where f : R
n
R, g : R
n
R
m
, h : R
n
R

, for some positive integers n, m and .


Remark 1.1 To t the opamp design problem into formulation (1.11) we had to pick one
of the design specications as objective (to be minimized). Intuitively more appealing would
be some kind of multiobjective optimization problem.
Example 1.2 Design of a p.i.d controller (proportional - integral - derivative)
The scalar plant G(s) is to be controlled by a p.i.d. controller (see Figure 1.1). Again, the
structure of the controller has already been chosen; only the values of three parameters have
to be determined (x = [x
1
, x
2
, x
3
]
T
).
R(s)
E(x,s) Y(x,s)
T(x,s)

+
x + x s +
1 2
3
s
x
G(s)

Figure 1.1:
Suppose the specications are as follows
low value of the ISE for a step input (ISE = integral of the square of the dierence
(error) between input and output, in time domain)
enough stability
short rise time, settling time, low overshoot
We decide to minimize the ISE, while keeping the Nyquist plot of T(x, s) outside some
forbidden region (see Figure 1.2) and keeping rise time, settling time, and overshoot under
given values.
The following constraints are also specied.
10 x
1
10 , 10 x
2
10 , .1 x
3
10
Exercise 1.5 Put the p.i.d. problem in the form (1.6), i.e., specify f, g
i
,
i
,
i
.
6 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
1.1 Some Examples
parabola
T(x,jw)
y=(a+1)x - b
2
w=w
w=0
(-1,0)
1
(t)
u(t)
T
r
T
s
T
y(t)
Figure 1.2:
T(x, j) has to stay outside the forbidden region [0, ]
For a step input, y(x, t) is desired to remain between l(t) and u(t) for t [0, T]
Example 1.3 Consider again a plant, possibly nonlinear and time varying, and suppose we
want to determine the best control u(t) to approach a desired response.
x = F(x, u, t)
y = G(x, t)
We may want to determine u() to minimize the integral
J(u) =
_
T
0
(y
u
(t) v(t))
2
dt
where y
u
(t) is the output corresponding to control u() and v() is some reference signal.
Various features may have to be taken into account:
Constraints on u() (for realizability)
piecewise continuous
[u(t)[ u
max
t
T may be nite or innite
x(0), x(T) may be free, xed, constrained
The entire state trajectory may be constrained (x()), e.g., to keep the temperature
reasonable
One may require a closed-loop control, e.g., u(t) = u(x(t)). It is well known that
such feedback control systems are much less sensitive to perturbations and modeling
errors.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 7
Motivation and Scope
Unlike Example 1.1 and Example 1.2, Example 1.3 is an optimal control problem. Whereas
discrete-time optimal control problems can be solved by classical optimization techniques,
continuous-time problems involve optimization in innite dimension spaces (a complete
waveform has to be determined).
To conclude this section we now introduce the class of problems that will be studied in this
course. Consider the abstract optimization problem
(P) minf(x) [ x S
where S is a subset of a vector space X and where f : X R is the cost or objective
function. S is the feasible set. Any x in S is a feasible point.
Denition 1.1 A point x is called a (strict) global minimizer for (P) if x S and
f( x) f(x) x S
(<) (x S, x ,= x)
Assume now X is equipped with a norm.
Denition 1.2 A point x is called a (strict) local minimizer for (P) if x S and > 0
such that
f( x) f(x) x S B( x, )
(<) (x S B( x, ), x ,= x)
8 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
1.2 Scope of the Course
1.2 Scope of the Course
1. Type of optimization problems considered
(i) Finite-dimensional
unconstrained
equality constrained
inequality [and equality] constrained
linear, quadratic programs, convex problems
multiobjective problems
discrete optimal control
(ii) Innite-dimensional
calculus of variations (no control signal) (old: 1800)
optimal control (new: 1950s)
Note: most types in (i) can be present in (ii) as well.
2. Results sought
Essentially, solve the problem. The steps are
conditions of optimality (simpler characterization of solutions)
numerical methods: solve the problem, generally by solving some optimality con-
dition or, at least, using the insight such conditions provide.
sensitivity: how good is the solutions in the sense of what if we didnt solve
exactly the right problem?
( duality: some transformation of the original problem into a hopefully simpler
optimization problem)
Copyright c _19932011, Andre L. Tits. All Rights Reserved 9
Motivation and Scope
10 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
Chapter 2
Linear Optimal Control: Some
(Reasonably) Simple Cases
References: [24, 1].
Consider the linear control system
x(t) = A(t)x(t) + B(t)u(t), x(t
0
) = x
0
(2.1)
where x(t) R
n
, u(t) R
m
and A() and B() are matrix-valued functions. Suppose A(),
B() and u() are continuous. Then (2.1) has the unique solution
x(t) = (t, t
0
)x
0
+
t
_
t
0
(t, )B()u()d
where the state transition matrix satises the homogeneous dierential equation

t
(t, t
0
) = A(t)(t, t
0
)
with initial condition
(t
0
, t
0
) = I.
Further, for any t
1
, t
2
, the transition matrix (t
1
, t
2
) is invertible and
(t
1
, t
2
)
1
= (t
2
, t
1
).
Throughout these notes, though, we will typically merely assume that the components of
the control function u are piecewise continuous. The reason for this is that, as will become
clear later on (e.g., see bang-bang control), in many important cases, optimal controls are
naturally discontinuous, i.e., the problem has no solution if minimization is carried out
over the set of continuous function.
Denition 2.1 A function u : R R
m
is piecewise continuous (or p.c.) if it is right-
continuous and, given any a, b R with a < b, it is continuous on [a, b] except for possibly
nitely many points of discontinuity.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 11
Linear Optimal Control: Some (Reasonably) Simple Cases
Note that some authors do not insist on right-continuity. The reason we do is that
without such requirement there typically will not be a unique optimal controlchanging the
value of an optimal control at, say, a single point

t does not aect is optimality. (A rst
instance of this appears in the next section, in connection with equations (2.9)-(2.10).
Also note that, if u is allowed to be discontinuous, solutions to dierential equation 2.1)
must be understood as satisfying it almost everywhere.
Thus, unless otherwise specied, in the sequel, the set of admissible controls will be the
set | is dened by
| := u : [t
0
, t
1
] R
m
, p.c..
2.1 Free terminal state, unconstrained, quadratic cost
2.1.1 Finite horizon
Consider the optimal control problem
minimize J(u)
1
2
_
t
1
t
0
_
x(t)
T
L(t)x(t) + u(t)
T
u(t)
_
dt +
1
2
x(t
1
)
T
Qx(t
1
)
subject to x(t) = A(t)x(t) + B(t)u(t) (2.2)
x(t
0
) = x
0
, (2.3)
u |, (2.4)
where x(t) R
n
, u(t) R
m
and A(), B() and L() are matrix-valued functions, minimiza-
tion is with respect to u and x. The initial and nal times t
0
and t
1
are given, as is the initial
state x
0
. The mappings A(), B(), and L(), dened on the domain [t
0
, t
1
], are assumed to
be continuous. Without loss of generality, L and Q are assumed symmetric.
The problem just stated is, in a sense, the simplest continuous-time optimal control
problem. Indeed, the cost function is quadratic and the dynamics linear, and there are no
constraints (except for the prescribed initial state). While a linear cost function may be
even simpler than a quadratic one, in the absence of (implicit or explicit) constraints on the
control, such problem would have no solution (unless the cost function is a constant).
In fact, the problem is simple enough that it can be solved without much advanced
mathematical machinery, by simply completing the squares. Doing this of course requires
that we add to and subtract from J(u) a quantity that involves x and u. But doing so would
likely modify our problem! The following fundamental lemma gives us a key to resolving
this conundrum.
Lemma 2.1 (Fundamental Lemma) Let A(), B() be continuous matrix-value functions and
K() = K
T
() be a matrix-valued function, with continuously dierentiable entries. Suppose

K(t) exists on [t
0
, t
1
]. Then, if x(t) and u(t) are related by
x(t) = A(t)x(t) + B(t)u(t), (2.5)
it holds
x(t
1
)
T
K(t
1
)x(t
1
) x(t
0
)
T
K(t
0
)x(t
0
)
12 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
2.1 Free terminal state, unconstrained, quadratic cost
=
_
t
0
t
1
_
x(t)
T
(

K(t) + A
T
(t)K(t) + K(t)A(t))x(t) + 2x(t)
T
K(t)B(t)u(t)
_
dt (2.6)
Proof. Because x()
T
K()x() is continuously dierentiable (or just piecewise continuously
dierentiable)
x(t
1
)
T
K(t
1
)x(t
1
) x(t
0
)
T
K(t
0
)x(t
0
) =
t
1
_
t
0
d
dt
x(t)
T
K(t)x(t) dt
=
t
1
_
t
0
( x(t)
T
K(t)x(t) + x(t)
T

K(t)x(t) + x(t)
T
K(t) x(t))dt
and the claim follows if one substitutes for x(t) the right hand side of (2.5).
Note that, while the integral in this lemma involves the paths x(t), u(t), t [t
0
, t
1
], its value
depends only on the end points of the trajectory x(), i.e., this integral is path independent.
Using Lemma 2.1 we can now complete the square without aecting J(u). First, for any
K() satisfying the assumptions, the lemma yields
x
T
0
K(t
0
)x
0
=
_
t
0
t
1
_
x(t)
T
(

K(t) + A
T
(t)K(t) + K(t)A(t))x(t) + 2x(t)
T
K(t)B(t)u(t)
_
dt x(t
1
)
T
K(t
1
)x(t
1
).
Now, with reference to our optimal control problem, select K() to satisfy K(t
1
) = Q, and
dene

J(u) = J(u)
1
2
x
T
0
K(t
0
)x
0
.
Note that because K() is selected independently of u, minimizing

J(u) is equivalent to
minimizing J(u), i.e., they have the same minimizer(s). We obtain

J(u) =
_
t
0
t
1
_
1
2
x(t)
T
(L +

K + A
T
K + KA)x +
1
2
u(t)
T
u + x(t)
T
KBu(t)
_
dt. (2.7)
Now, select K(t) to satisfy the dierential equation

K(t) = A
T
(t)K(t) K(t)A(t) L(t) + K(t)B(t)B
T
(t)K(t) (2.8)
while still imposing the terminal condition
K(t
1
) = Q.
Equation (2.8) is an instance of a dierential Riccati equation. Its right-hand side is quadratic
in the unknown K. We postpone the discussion of existence and uniqueness of a solution to
this equation. The following scalar example shows that this is an issue indeed.
Example 2.1 (scalar Riccati equation)
Copyright c _19932011, Andre L. Tits. All Rights Reserved 13
Linear Optimal Control: Some (Reasonably) Simple Cases
Consider the case of scalar, time-independent values a = 0, b = 1, l = 1, q = 0. The
corresponding Riccati equation is

k = 1 + k
2
, k(T) = 0
We get
atan(k(t)) atan(k(T)) = t T,
yielding
k(t) = tan (t T),
with a nite escape time at t = T

2
.
For the time being, we assume that a solution exists on [t
0
, t
1
], and we denote its value
at time t by
K(t) = (t, Q, t
1
)
From (2.7) and (2.8), we get

J(u) =
1
2
t
1
_
t
0
|B
T
(t)(t, Q, t
1
)x(t) + u(t)|
2
dt. (2.9)
Since u is free, J(u) attains its minimal value when the rst term is set to zero, yielding the
closed-loop optimal control law
u

(t) = B
T
(t)(t, Q, t
1
)x(t). (2.10)
Remark 2.1 By closed-loop it is meant that the right-hand side of (2.10) does not depend
on the intial state x
0
nor on the initial time t
0
, but only on the current state and time. Such
formulations are of major practical importance: If, for whatever reason (modeling errors,
perturbations) the state a time t is not what it was predicted to be (when the optimal
control u

() was computed, at time t


0
), (2.10) is still optimal over the remaining time
interval (assuming no modeling errors or perturbations between times t and t
1
).
To formally verify that u

is optimal, simply substitute (2.10) into (2.1) and note that the
resulting autonomous system has a unique solution x, which is continuous. Plugging this
solution into (2.10) yields an open-loop expression for u

, which also shows that it is


continuous, hence belongs to |.
In view of (2.9), the optimal value is given by
V (x
0
, t
0
) := J(u

) =
1
2
x
T
0
(t
0
, Q, t
1
)x
0
,
where V is known as the value function.
Now suppose that, starting from x(t
0
) at time t
0
, the state reaches x() at time
(t
0
, t
1
). The remaining portion of the minimal cost is the minimum, over u |, subject to
x = Ax + Bu with x() xed, of
J

(u) :=
_
t
1

_
x(t)
T
L(t)x(t) + u(t)
T
u(t)
_
dt + x()
T
(, Q, t
1
)x() (2.11)
=
_
t
1

|B(t)
T
(, Q, t
1
)x(t) + u(t)|
2
dt + x()
T
(, Q, t
1
)x(), (2.12)
14 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
2.1 Free terminal state, unconstrained, quadratic cost
i.e, the cost-to-go is
J

(u

) = x()
T
(, Q, t
1
)x().
Hence, the cost-to-go from any time t < t
1
and state x R
n
is
V (x, t) = J
t
(u

) =
1
2
x
T
(t, Q, t
1
)x. (2.13)
Remark 2.2 We have not made any positive deniteness (or semi-deniteness) assumption
on L(t) or Q. The key assumption we have made is that the stated Riccati equation has
a solution (t, Q, t
1
). Below we investigate conditions (in particular, on L and Q) which
insure that this is the case. At this point, note that, if L(t) _ 0 for all t and Q _ 0, then
J(u) 0 for all u |, and expression (2.13) of the cost-to-go implies that (t, Q, t
1
) _ 0
whenever it exists.
Exercise 2.1 Investigate the case of the more general cost function
J(u) =
t
1
_
t
0
_
1
2
x(t)
T
L(t)x(t) + u(t)
T
S(t)x(t) +
1
2
u(t)
T
R(t)u(t)
_
dt,
where R(t) _ 0 for all t. Hints: (i) dene a new inner product on |; (ii) let v(t) =
u(t) + M(t)x(t) with M(t) judiciously chosen.
Returning to the question of existence/uniequeness of the solution to the dierential Riccati
equation, rst note that the right-hand side of (2.8) is Lipschitz continuous over bounded
subsets of R
n
, and that this implies that, for any given Q, there exists < t
1
such that
the solution exists, and is unique, in [, t
1
]. Now, for t [, t
1
], dene p(t) = (t, Q, t
1
)x(t),
so that the optimal control law (2.10) satises u

(t) = B
T
(t)p(t). Then x and p together
satisfy the the linear system
_
x(t)
p(t)
_
=
_
A(t) B(t)B
T
(t)
L(t) A
T
(t)
_ _
x(t)
p(t)
_
(2.14)
evolving in R
2n
.
Exercise 2.2 Verify that x and p satisfy (2.14).
We rst note that the Riccati equation (2.14) can be solved via system (2.14).
Theorem 2.1 Let (, ) be the 2n 2n state transition matrix for system (2.14) and let
_
X(t)
P(t)
_
= (t, t
1
)
_
I
Q
_
Then
(t, Q, t
1
) = P(t)X(t)
1
solves (2.14) for t [, t
1
], for any < t
1
such that X(t)
1
exists on [, t
1
].
Copyright c _19932011, Andre L. Tits. All Rights Reserved 15
Linear Optimal Control: Some (Reasonably) Simple Cases
Proof. Just plug in and use the fact that
_

X(t)

P(t)
_
=
_
A(t) B(t)B
T
(t)
L(t) A
T
(t)
_ _
X(t)
P(t)
_
.
Note that, since X(t
1
) = I, by continuity of the state transition matrix, X(t) is invertible
for t close enough to t
1
, conrming that the Riccati equation has a solution for t close to
t
1
. We know that this solution is unique. Since clearly the transpose of a solution to the
Riccati equation is also a solution, this unique solution must be symmetric, as required in
the path-independent lemma. Also, the only way the solution can disappear is if X(t)
becomes singular, but this in itself does not imply that (t, Q, t
1
) must grow without bound.
We now show that, when (t, Q, t
1
) disappears, it indeed must rst grow without bound.
The following classical results will be instrumental.
Fact. (See, e.g., [12, Chapter 1, Theorem 2.1].) Let : R R R
n
be continuous, and
Lipschitz continuous in its rst argument on every bounded set. Then for every x
0
R
n
,
t
0
R, there exists t
1
, t
2
R, with t
0
(t
1
, t
2
), such that the dierential equation x =
(x(t), t) has a solution x(t) in (t
1
, t
2
). Furthermore, this solution is unique. Finally, suppose
there exists a compact set S R
n
, with x
0
S, that enjoys the following property: For
every t
1
, t
2
such that t
0
(t
1
, t
2
) and the solution x(t) exists for all t (t
1
, t
2
), x(t) belongs
to S for all t (t
1
, t
2
). Then the solution x(t) exists and is unique for all t R.
Lemma 2.2 Let

t = inf : (t, Q, t
1
) exists t [, t
1
]. If

t > , then |(, Q, t
1
)| is
unbounded on (

t, t
1
].
Proof. Let
(K, t) = A(t)
T
K KA(t) + KB(t)B
T
(t)K L(t),
so that (2.8) can be written

K(t) = (K(t), t), K(t


1
) = Q,
where is continuous and is Lipschitz continuous in its rst argument on every bounded set.
Proceeding by contradiction, let S be a compact set containing (t, Q, t
1
) : t (

t, t
1
].
The claim is then an immediate consequence of the previous Fact.
Theorem 2.2 Suppose L(t) is positive semi-denite for all t and Q is positive semi-denite.
Then (t, Q, t
1
) exists t t
1
.
Proof. Again, let

t = inf : (t, Q, t
1
) exists t [, t
1
], so that (, Q, t
1
) exists in
(

t, t
1
]. Proceeding by contradiction, suppose that

t is nite. We will show that (t, Q, t
1
)
is bounded as t ranges over (

t, t
1
]; this contradiction with Lemma 2.2 will prove the claim.
Thus, let (

t, t
1
]. For any x R
n
, using the positive deniteness assumption on L(t) and
Q, we have
x
T
(, Q, t
1
)x = min
u|
t
1
_

(u(t)
T
u(t) + x(t)
T
L(t)x(t))dt + x(t
1
)
T
Qx(t
1
) 0 (2.15)
16 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
2.1 Free terminal state, unconstrained, quadratic cost
where x(t) satises (2.2) with initial condition x() = x. Letting x(t) be the solution to
(2.2) corresponding to u(t) = 0 t and x() = x, i.e., x(t) = (t, )x, and using (2.15), we
can write
0 x
T
(, Q, t
1
)x
t
1
_

x(t)
T
L(t) x(t)dt + x(t
1
)
T
Q x(t
1
= x
T
F()x (2.16)
with
F() =
_
t
1

(t, )
T
L(t)(t, )dt + (t
1
, )
T
Q(t
1
, ),
a continuous function. Since this holds for all x R
n
and since (t, Q, t
1
) is symmetric, it
follows that, using, e.g., the spectral norm,
|(, Q, t
1
)| |F()| (

t, t
1
].
Since F() is continuous on (, t
1
], hence bounded on [

t, t
1
], a compact set, it follows that
(, Q, t
1
) is bounded on (

t, t
1
], completing the proof by contradiction.
Exercise 2.3 Prove that, if A = A
T
and F = F
T
, and 0 x
T
Ax x
T
Fx for all x, then
|A|
2
|F|
2
, where | |
2
denotes the spectral norm. (I.e., |A|
2
= max|Ax|
2
: |x|
2
=
1.)
Thus, when L(t) is positive semi-denite for all t and Q is positive semi-denite, our problem
has a unique optimal control given by (2.10).
Pontryagins Principle
In connection with cost function J with u
T
u generalized to u
T
Ru (although we will still
assume R = I for the time being), let H : R
n
R
n
R
m
R R be given by
H(, , v, ) =
1
2
v
T
Rv +
1
2

T
L() +
T
(A() + B()v),
and let 1 : R
n
R
n
R R be dened by
1(, , ) = min
vR
m
H(, , v, ),
where clearly (when R = I), the minimum is achieved when v = B
T
(). Function H is
the pre-Hamiltonian,
1
(sometimes called control Hamiltonian or pseudo-Hamiltonian) and 1
the Hamiltonian (or true Hamiltonian). Thus
1(, , t) =
1
2

T
L(t) +
T
A(t)
1
2

T
B(t)B
T
(t)
=
1
2
_

_
T
_
L(t) A
T
(t)
A(t) B(t)B
T
(t)
_ _

_
.
1
This terminology is borrowed from P.S. Krishnaprasad
Copyright c _19932011, Andre L. Tits. All Rights Reserved 17
Linear Optimal Control: Some (Reasonably) Simple Cases
The gradient of 1 with respect to the rst 2n arguments is given by
1(, , t) =
_
L(t) A
T
(t)
A(t) B(t)B
T
(t)
_ _

_
.
System (2.14) can be equivalently written in the canonical form
x(t) =
p
1(x(t), p(t), t) =
p
H(x(t), p(t), u

(t), t)
p(t) =
x
1(x(t), p(t), t) =
x
H(x(t), p(t), u

(t), t).
i.e.,
z(t) = J1(x(t), p(t), t) = JH(x(t), p(t), u

(t), t) (2.17)
where z(t) =
_
x(t)
p(t)
_
and J =
_
0 I
I 0
_
. The auxiliary conditions can be written
x(t
0
) = x
0
, p(t
1
) = Qx(t
1
),
yielding a two-point boundary-value problem. Note that, equivalently, p(t
1
) = (x(t
1
)),
where is the terminal cost (x) =
1
2
x
T
Qx. Also, the optimal cost J(u

) can be equivalently
expressed at x(t
0
)
T
p(t
0
).
This is one instance of Pontryagins Principle; see Chapter 5 for more details. In general,
however, 1 may be nonsmooth (due to the min operation), so the expressions involving
its gradients may not be valid.
Remark 2.3 Along trajectories of (2.14),
d
dt
1(x(t), p(t), t) = 1(x(t), p(t), t)
T
z(t) +

t
1(x(t), p(t), t)
=

t
1(x(t), p(t), t)
since
1(x(t), p(t), t)
T
z(t) = 1(x(t), p(t), t)
T
J1(x(t), p(t), t)
= 0 (since J
T
= J).
In particular, if A, B and L do not depend on t, 1(x(t), p(t), t) is constant along trajectories
of (2.14).
2.1.2 Innite horizon, LTI systems
We now turn our attention to the case of innite horizon (t
1
= ). To simplify the analysis,
we also assume that A, B and L are constant. We also simplify the notation by translating
the origin of time to the intial time t
0
, i.e., by letter t
0
= 0. Assuming (as above) that
L = L
T
0, we write L = C
T
C, so that the problem can be written as
18 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
2.1 Free terminal state, unconstrained, quadratic cost
minimize J(u) =

_
0
_
y(t)
T
y(t) + u(t)
T
u(t)
_
dt
subject to
x = Ax + Bu
y = Cx
x(0) = x
0
Note that y is merely some linear image of x, and need not be a physical output. For
example, it could be an error output.
Consider the dierential Riccati equation (2.8) with our constant A, B, and L and, in
agreement with the notation used in the previous section, denote by (t, Q, ) the value at
time t of the solution to this equation that takes some value Q at time . Since L is positive
semi-denite, such solution exists for all t and (with t ) and is unique, symmetric and
positive semi-denite. It is intuitively reasonable to expect that, for xed t, as goes to ,
the optimal feedback law u(t) = B
T
(t, 0, )x(t) for the nite-horizon problem with cost
function
J

(u) =

_
0
(u(t)
T
u(t) + x(t)
T
C
T
Cx(t))dt .
will tend to an optimal feedback law for our innite horizon problem. Since the optimal
cost-to-go is x(t)
T
(t, 0, )x(t), it is also reasonable (or, at least, tempting) to expect that
(t, 0, ) itself will converge, as , to some matrix

independent of t (since the


time-to-go will approach at every time t and the dynamics is time-invariant), satisfying
the algebraic Riccati equation (since the derivative of

with respect to t is zero)


A
T

BB
T

+ C
T
C = 0 (ARE)
We will see that, under a mild assumption, introduced next, our intuition is correct.
Before proceeding further, let us gain some intuition concerning (ARE). In the scalar
case, if C ,= 0, the left-hand side is a downward parabola that intersects the vertical axis at
C
2
( 0); hence (ARE) has to real solutions, one nonnegative, the other nonpositive. When
n > 1, the number of solutions increases, most of them being indenite matrices (i.e., with
some positive eigenvalues and some negative eigenvalues). A (or the) positive semi-denite
solution will be the focus of our investigation.
When dealing with an innite-horizon problem, it is natural (if nothing else, for practical
reasons) to seek control laws that are stabilizing. (Note that optimality does not imply
stabilization: E.g., if C = 0, then the control u = 0 is obviously optimal, regardless of
whether or not the system is stable.) This prompts the following natural assumption:
Assumption. (A, B) is stabilizable.
In accordance with the intuition introduced above, we focus out attention on the matrix
() := (0, 0, ).
We rst show that (t) converges to

satisfying (ARE), then that the control law u(t) =


B
T

x(t) is optimal for our innite-horizon problem.


Copyright c _19932011, Andre L. Tits. All Rights Reserved 19
Linear Optimal Control: Some (Reasonably) Simple Cases
Lemma 2.3 As t , (t) converges to some symmetric, positive semi-denite matrix

.
Proof. Given x
0
R
n
we have, for 0,
x
T
0
()x
0
= min
u|
J

(u).
It follows that () 0 for t 0. Next, let u(t) = F x(t) be any stabilizing static state
feedback and let x be the corresponding solution of
x = (A+ BF)x
x(0) = x
0
,
i.e., x(t) = e
(A+BF)t
x
0
. Then, since A+ BF is stable,
x
T
0
()x
0

_
0
x()
T
C
T
C x()d + u()
T
u() = x
T
0
M x
0
0
where
M =

_
0
e
(A+BF)
T
t
(C
T
C + F
T
F)e
(A+BF)t
dt
is well dened and independent of . Now, for all 0, we have
x
T
0
()x
0
= min
u|

_
0
x()
T
C
T
Cx()d + u()
T
u(). (2.18)
Non-negative deniteness of C
T
C implies that x
T
0
()x
0
is monotonically nondecreasing as
increases. Since it is bounded, it must converge for any x
0
.
2
Using the fact that () is
symmetric it is easily shown (see Exercise 2.4) that it converges, i.e., for some symmetric
matrix

,
lim

() =

.
Symmetry and positive semi-deniteness are inherited from ().
Exercise 2.4 Prove that convergence of x
T
0
()x
0
for arbitrary x
0
implies convergence of
().
Now note that, due to time-invariance of the dierential Riccati equation, for all t ,
( t) = (0, 0, t) = (t, 0, )
so that, under our stabilizability assumption, whave have that, for every t,

= lim

(t, 0, ).
The proof of the following lemma is adapted from that in [14] (p.232), which is due to
Kalman.
2
It cannot be concluded at this stage that its derivative converges to zero. In particular, Barbalats
Lemma cant be applied without rst showing that () has a uniformly continuous derivative.
20 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
2.1 Free terminal state, unconstrained, quadratic cost
Lemma 2.4

satises (ARE).
Proof. Because the DRE solution with value 0 at time is equal to the DRE solution with
value (0, 0, ) at time 0, we get

= lim

(t, (0, 0, ), 0).


Now, in view of Theorem 2.1, the solution of the Riccati equation depends continuously on
the initial condition. Hence

= (t, lim

(0, 0, ), 0) = (t,

, 0)
which proves our claim. Now, since

is constant, its solving the DRE implies that it


solves the ARE, and this completes the proof.
Now note that, for any u | and 0, we have
x
T
0
()x
0
= x
T
0
(0, 0, )x
0
= min
u|
J

(u) J

( u) J( u).
Letting we obtain
x
T
0

x
0
J( u) u |
implying that
x
T
0

x
0
inf
u|
J(u). (2.19)
Finally, we construct a control u

which attains the cost value x


T
0

x
0
, proving optimality
of u

and equality in (2.19). For this, note that the constant matrix

, since it satises
(ARE), automatically satises the corresponding dierential Riccati equation. Using this
fact and making use of the Fundamental Lemma, we obtain, analogously to (2.9),
J

(u) =
_

0
|u(t) + B
T

x(t)|
2
2
x()
T

x() + x
T
0

x
0
0..
Substituting the feedback control law u

= B
T

x and using the fact that

is positive
semi-denite yields
J

(u

) x
T
0

x
0
0.
Taking limits as yields
J(u

) = lim

(u

) x
T
0

x
0
.
Hence we have proved the following result.
Theorem 2.3 Suppose (A, B) is stabilizable. Then

solves (ARE), the control law u

=
B
T

x is optimal, and
J(u

) = x
T
0

x
0
.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 21
Linear Optimal Control: Some (Reasonably) Simple Cases
This solves the innite horizon LTI problem. Note however that it is not guaranteed that
the optimal control law is stabilizing. For example, consider the extreme case when C = 0
(in which case

= 0) and the system is open loop unstable. It seems clear that this is
due to unstable modes not being observable through C. We now show that, indeed, under a
detectability assumption, the optimal control law is stabilizing.
Theorem 2.4 Suppose (A, B) is stabilizable and (C, A) detectable. Then, if K _ 0 solves
(ARE), ABB
T
K is stable; in particular, ABB
T

is stable.
Proof. (ARE) can be rewritten
(ABB
T
K)
T
K + K(A BB
T
K) = KBB
T
K C
T
C. (2.20)
Proceed now by contradiction. Let , with Re 0, and v ,= 0 be such that
(ABB
T
K)v = v. (2.21)
Multiplying (2.20) on the left by v

and on the right by v we get


2(Re)(v

Kv) = |B
T
Kv|
2
|Cv|
2
.
Since the left-hand side is non-negative (since K _ 0) and the right-hand side non-positive,
both sides must vanish. Thus (i) Cv = 0 and, (ii) B
T
Kv = 0 which together with (2.21),
implies that Av = v. Since Re 0, this contradicts detectability of (C, A).
Corollary 2.1 If (A, B) is stabilizable and (C, A) is detectable, then the optimal control law
u = B
T

x is stabilizing.
Exercise 2.5 Suppose (A, B) is stabilizable. Then, if

x
0
= 0 (equivalently, x
T
0

x
0
=
0) for some x
0
, then x
0
belongs to the unobservable subspace. In particular, if (C, A) is
observable, then

> 0. [Hint: Use the fact that J(u

) = x
T
0

x
0
.]
Finally, we discuss how (ARE) can be solved directly (without computing the limit of
(t)). In the process, we establish that, under stabilizability of (A, B) and detectability of
(C, A),

is that unique stabilizing solution of (ARE), hence the unique symmetric positive
semi-denite solution of (ARE).
Computation of

Consider the Hamiltonian matrix (see (2.14))


H =
_
A BB
T
C
T
C A
T
_
,
and let K
+
be a stabilizing solution to (ARE), i.e., A BB
T
K
+
is stable. Let
T =
_
I 0
K
+
I
_
.
22 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
2.1 Free terminal state, unconstrained, quadratic cost
Then
T
1
=
_
I 0
K
+
I
_
.
Now (elementary block column and block row operations)
T
1
HT =
_
I 0
K
+
I
_ _
A BB
T
L A
T
_ _
I 0
K
+
I
_
=
_
I 0
K
+
I
_ _
ABB
T
K
+
BB
T
L A
T
K
+
A
T
_
=
_
ABB
T
K
+
BB
T
0 (A BB
T
K
+
)
T
_
since K
+
is a solution to (ARE). It follows that
(H) = (A BB
T
K
+
) ((A BB
T
K
+
)),
where () denotes the spectrum (set of eigenvalues). Thus, if (A, B) is stabilizable and
(C, A) detectable (so such solution K
+
exists), H cannot have any imaginary eigenvalues:
It must have n eigenvalues in C

and n eigenvalues in C
+
. Furthermore the rst n columns
of T form a basis for the stable invariant subspace of H, i.e., for the span of all generalized
eigenvectors of H associated with stable eigenvalues (see, e.g., Chapter 13 of [27] for more
on this).
Now suppose (ARE) has a stabilizing solution, and let
_
S
11
S
21
_
be any basis for the stable
invariant subspace of H, i.e., let
S =
_
S
11
S
12
S
21
S
22
_
be any invertible matrix such that
S
1
HS =
_
X Z
0 Y
_
for some X, Y, Z such that (X) C

and (Y ) C
+
. (Note that (H) = (X) (Y ).)
Then it must hold that, for some non-singular R

,
_
S
11
S
21
_
=
_
I
K
+
_
R

.
It follows that S
11
= R

and S
21
= K
+
R

, thus
K
+
= S
21
S
1
11
.
We have thus proved the following.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 23
Linear Optimal Control: Some (Reasonably) Simple Cases
Theorem 2.5 Suppose (A, B) is stabilizable and (C, A) is detectable, and let K
+
be a sta-
bilizing solution of (ARE). Then K
+
= S
21
S
1
11
, where
_
S
11
S
21
_
is any basis for the stable
invariant subspace of H. In particular, S
21
S
1
11
is symmetric and there is exactly one stabi-
lizing solution to (ARE), namely

(which is also the only positive semi-denite solution).


Exercise 2.6 Given J =
_
0 I
I 0
_
, any real matrix H that satises J
1
HJ = H
T
is
said to be Hamiltonian. Show that if H is Hamiltonian and is an eigenvalue of H, then
also is.
2.2 Free terminal state, constrained control values, lin-
ear cost
Consider the linear system
(S) x(t) = A(t)x(t) + B(t)u(t), t t
0
where A(), B() are assumed to be piecewise continuous In contrast with the situation
considered in the previous section, the control u is constrained to take values in a xed set
U R
m
. Accordingly, the set of admissible controls, still denoted by | is now
| := u : [t
0
, t
1
] U, p.c..
Given this constraint on the values of u (set U), contrary to Chapter 2, a linear cost
function can now be meaningful. Accordingly, we start with the following problem.
Let c R
n
, c ,= 0, x
0
R
n
and let t
1
t
0
be a xed time. We are concerned with the
following problem. Find a control u

| so as to
minimize c
T
x(t
1
) s.t.
dynamics x(t) = A(t)x(t) + B(t)u(t) t
0
t t
1
initial condition x(t
0
) = x
0
nal condition x(t
1
) R
n
(no constraints)
control constraint u |
Remark 2.4 For any u |, the solution to the given linear dierential equation is well
dened and unique.
Notation: For any piecewise continuous u : [t
0
, t
f
] R
m
, any z R
n
and t
0
t
1
t
2

t
f
, let (t
2
, t
1
, z, u) denote the state at time t
2
if at time t
1
the state is z and control u is
applied, i.e., let
(t
2
, t
1
, z, u) = (t
2
, t
1
)z +
_
t
2
t
1
(t
2
, )B()u()d.
24 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
2.2 Free terminal state, constrained control values, linear cost
Also let
K(t
2
, t
1
, z) = (t
2
, t
1
, z, u) : u |.
This set is called reachable set.
Denition 2.2 The adjoint system to system x(t) = A(t)x(t) is given by
p(t) = A(t)
T
p(t).
Let (t, ), t
0
t t
f
be the state transition matrix for the given linear system, i.e.,
satises the dierential equation

t
(t, ) = A(t)(t, )
and the boundary condition
(t, t) = I
Then the state transition matrix (t, ) of the adjoint system is given by
(t, ) = (, t)
T
.
Indeed, it is readily checked that, if x = Ax, then x(t)
T
p(t) is constant.
Exercise 2.7 Prove that
d
dt

A
(t
0
, t) =
A
(t
0
, t)A(t).
Theorem 2.6 Let u

| and let
x

(t) = (t, t
0
, x
0
, u

), t
0
t t
f
.
Let p

(t) satisfy the adjoint equation:


p

(t) = A
T
(t)p

(t) t
0
t t
f
with nal condition
p

(t
f
) = c
Then u

is optimal if and only if


p

(t)
T
B(t)u

(t) = infp

(t)
T
B(t)v : v U (2.22)
(implying that the inf is achieved) for all t [t
0
, t
f
]. [Note that the minimization is now
over a nite dimension space.]
Proof. u

is optimal if and only if, for all u |


c
T
[(t
f
, t
0
)x
0
+
_
t
f
t
0
(t
f
, )B()u

()d] c
T
[(t
f
, t
0
)x
0
+
_
t
f
t
0
(t
f
, )B()u()d]
or, equivalently,
_
t
f
t
0
((t
f
, )
T
c)
T
B()u

()d
_
t
f
t
0
((t
f
, )
T
c)
T
B()u()d
Copyright c _19932011, Andre L. Tits. All Rights Reserved 25
Linear Optimal Control: Some (Reasonably) Simple Cases
As pointed out above, for p

(t) as dened,
p

(t) = (t
f
, t)
T
p

(t
f
) = (t
f
, t)
T
c
So that u

is optimal if and only i, for all u |,


_
t
f
t
0
p

()
T
B()u

()d
_
t
f
t
0
p

()
T
B()u()d
and the if direction of the theorem follows immediately. Suppose now u

| is optimal.
Let D be the nite subset of [t
0
, t
f
] where B() or u

is discontinuous. We show that, if u

is optimal, (2.22) is satised t , D. Indeed, if this is not the case, t

, D, v U s.t.
p

(t

)
T
B(t

)u

(t

) > p

(t

)
T
B(t

)v.
By continuity, this inequality holds in a neighborhood of t

, say [t t

[ < , with > 0.


Dene u | by
u(t) =
_
v [t t

[ < , t [t
0
, t
f
]
u

(t) otherwise
Then it follows that
_
t
f
t
0
p

(t)
T
B(t)u

(t)dt >
_
t
f
t
0
p

(t)
T
B(t) u(t)dt
which contradicts optimality of u

.
Corollary 2.2 For t
0
t

t t
1
,
p

(t)
T
x

(t) p

(t)
T
x x K(t, t

, x

(t

)) (2.23)
Exercise 2.8 Prove the corollary.
Because the problem under consideration has no integral cost, the pre-Hamiltonian H dened
in section 2.1.1 reduces to
H(, , v, ) =
T
(A() + B()v)
and the Hamiltonian 1 by
1(, , ) = infH(, , v, ) : v U
then (since p
T
A(t)x does not involve the variable u) condition (2.22) can be written as
H(t, x

(t), u

(t), p

(t)) = 1(t, x

(t), p

(t)) t (2.24)
This condition is known as Pontryagins Principle (the phrase maximum principle is more
commonly used and corresponds to the equivalent maximization of c
T
x(t
f
)). The previous
theorem states that, for linear systems with linear objective functions, Pontryagins Principle
provides a necessary and sucient condition of optimality.
26 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
2.3 Fixed terminal state, unconstrained, quadratic cost
Remark 2.5
1. Let : R
n
R be the terminal cost function, dened by (x) = c
T
x. Then
p

(t
1
) = (x

(t
1
)), just like in the case of the problem of section 2.1.
2. Linearity in u was not used. Thus the result applies to systems with dynamics of the
form
x(t) = A(t)x(t) + B(t, u(t))
where B is, say, a continuous function.
3. The optimal control for this problem is independent of x
0
! (The reason is clear from
the rst two equations in the proof.)
Exercise 2.9 Compute the optimal control u

for the following time-invariant data: t


0
= 0,
t
1
= 1, A = diag(1, 2), B = [1; 1], c = [2; 1]. Note that u

is not continuous! (Indeed, this


is an instance of a bang-bang control.)
Fact. Let A and B be constant matrices, and suppose there exists an optimal control
u

, with corresponding trajectory x

. Then m(t)

= 1(t, x

(t), p

(t)) is constant (i.e., the


Hamiltonian is constant along the optimal trajectory).
Exercise 2.10 Prove the fact under the assumption that U = [, ]. (The general case will
be considered in Chapter 5.)
Exercise 2.11 INCORRECT! Suppose that U is compact and suppose that there exists
an optimal control. Also suppose that B(t) vanishes at no more than nitely many points t.
Show that there exists an optimal control u

such that u

(t) belongs to the boundary of U for


all t.
Exercise 2.12 Suppose U = [, ], so that B(t) is an n 1 matrix. Suppose that A(t) = A
and B(t) = B are constant matrices and A has n distinct real eigenvalues. Show that there is
an optimal control u

and t
0

1
< . . .
n
= t
1
(n = dimension of x) such that u

(t) =
or on [t
i
, t
i+1
), 0 i n. [Hint: rst show that p

(t)
T
B =
1
exp(
1
t) +. . . +
n
exp(
n
t)
for some
i
,
i
R. Then use induction.]
2.3 Fixed terminal state, unconstrained, quadratic
cost
Background material for this section can be found in Appendix A.
Question: Given x
1
R
n
, t
1
> t
0
, does there exist u | such that, for system (2.1),
x(t
1
) = x
1
? If the answer to the above is yes, we say that x
1
is reachable from (x
0
, t
0
)
at time t
1
. If moreover this holds for all x
0
, x
1
R
n
then we say that the system (2.1) is
reachable on [t
0
, t
1
].
There is no loss of generality in assuming that x
1
= , as shown by the following exercise.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 27
Linear Optimal Control: Some (Reasonably) Simple Cases
Exercise 2.13 Dene x(t) := x(t) (t, t
1
)x
1
. Then x satises
d
dt
x(t) = A(t) x(t) + B(t)u(t).
Conclude that, under dynamics (2.1), u steers (x
0
, t
0
) to (x
1
, t
1
) if and only if it steers
(x
0
(t
0
, t
1
)x
1
, t
0
) to (, t
1
).
Since (t
0
, t
1
) is invertible, it follows that system (2.1) is reachable on [t
0
, t
1
] if and only if
it is controllable on [t
0
, t
1
], i.e., if and only if, given x
0
, there exists u | that steers (x
0
, t
0
)
to (, t
1
).
Note. Equivalence between reachability and controllability (to the origin) does not hold in
the discrete-time case, where controllability is a weaker property than reachability.
Now controllability to (, t
1
) from (, t
0
), for some , is equivalent to solvability of the
equation (in u |):
(t
1
, t
0
) +
t
1
_
t
0
(t
1
, )B()u()d = .
Equivalently (multiplying on the left by the non-singular matrix (t
0
, t
1
)), (, t
1
) can be
reached from (, t
0
) if there exists u | such that
=
t
1
_
t
0
(t
0
, )B()u()d.
If is indeed reachable at time t
1
from (, t
0
), we might want want to steer (, t
0
) while
spending the least amount energy, i.e., while minimizing
t
1
_
t
0
u(t)
T
u(t)dt.
It turns out that this optimal control problem can be solved nicely by making of the
linear algebra framework reviewed in Appendix A. Indeed, we have here a linear least-
squares problem over the inner-product vector space of continuous functions | = C[t
0
, t
1
]
m
.
To see this, note that L : | R
n
dened by
L(u) :=
t
1
_
t
0
(t
0
, )B()u()d
is a linear map, and that , : | | is an inner product. Clearly, is reachable at time
t
1
from (, t
0
) if and only if 1(L), so that (2.1) is controllable on [t
0
, t
1
] if and only if
1(L) = R
n
. And our xed endpoint minimum energy control problem becomes
(FEP) minimize u, u subject to L(u) = , u |.
It follows that the unique optimal control is given by the unique u 1(L

) satisfying Lu = ,
i,.e., L

, where L

is the Moore-Penrose pseudo-inverse of L, associated with inner product


, . Specically,
u = L

y,
28 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
2.3 Fixed terminal state, unconstrained, quadratic cost
where L

is the adjoint of L for inner product , , and where y is any point in R


n
satisfying
LL

y =
(and such points do exist). It is shown in Exercise A.44 of Appendix A that L

is given by
(L

x)(t) = B
T
(t)
T
(t
0
, t)x
which yields
LL

x =
t
1
_
t
0
(t
0
, )B()(L

x)()d
=
t
1
_
t
0
(t
0
, )B()B
T
()
T
(t
0
, )d x, x,
i.e., LL

: R
n
R
n
is given by
LL

=
t
1
_
t
0
(t
0
, t)B(t)B
T
(t)
T
(t
0
, t)dt =: W(t
0
, t
1
).
Since 1(L) = 1(LL

), is reachable at t
1
from (, t
0
) if and only if
1(W(t
0
, t
1
)) .
The matrix W(t
0
, t
1
) has entries
(W(t
0
, t
1
))
ij
=
_
(
i
(t
0
, )B())
T
_
T
(
j
(t
0
, )B())
T
,
where , is again the L
2
inner product, i.e., W(t
0
, t
1
) is the Gramian matrix (or
Gram matrix, or Gramian) associated with the vectors (
1
(t
0
, )B())
T
, . . . , (
n
(t
0
, )B())
T
,
(=B()
T
(
1
(t
0
, ))
T
, . . . , B()
T
(
n
(t
0
, ))
T
), which are the columns of the matrix B()
T
((t
0
, ))
T
).
It is known as the controllability Gramian.
It is invertible if and only if 1(L) = R
n
, i.e., if and only if the system is controllable on
[t
0
, t
1
]. Suppose this is the case and let
(x, t) := x (t, t
1
)x
1
,
so that dened earlier is given by
= (x(t
0
), t
0
).
The minimum energy control that steers (x
0
, t
0
) to (x
1
, t
1
) is then given by
u

= L

(LL

)
1
(x(t
0
), t
0
)
i.e.
u

(t) = B
T
(t)
T
(t
0
, t)W(t
0
, t
1
)
1
(x(t
0
), t
0
) (2.25)
Copyright c _19932011, Andre L. Tits. All Rights Reserved 29
Linear Optimal Control: Some (Reasonably) Simple Cases
and the corresponding energy is given by
u

, u

= (x(t
0
), t
0
)
T
W(t
0
, t
1
)
1
(x(t
0
), t
0
).
Note that, as expressed in (2.25), u

(t) depends explicitly, through (x


0
, t
0
), on the initial
state x
0
and initial time t
0
. Consequently, if between t
0
and the current time t, the state
has been aected by an external perturbation, u

as expressed by (2.25) is no longer optimal


(minimum energy) over the remaining time interval [t, t
1
]. Let us address this issue. At time
t
0
, we have
u

(t
0
) = B
T
(t
0
)
T
(t
0
, t
0
)W(t
0
, t
1
)
1
(x(t
0
), t
0
)
= B
T
(t
0
)W(t
0
, t
1
)
1
(x(t
0
), t
0
) .
Intuitively, this must hold independently of the value of t
0
, i.e., for the problem under
consideration, Bellmans Principle of Optimality holds: independently of the initial state (at
t
0
), for u

to be optimal for (FEP), it is necessary that u

applied from the current time


t t
0
up to the nal time t
1
, starting at the current state x(t), be optimal for the remaining
problem, i.e., for the objective function
_
t
1
t
u()
T
u()d.
Specically, given x R
n
and t [t
0
, t
1
] such that x
1
is reachable at time t
1
from (x, t),
denote by P(x, t; x
1
, t
1
) the problem of determining the control of least energy that steers
(x, t) to (x
1
, t
1
), i.e., problem (FEP) with (x, t) replacing (x
0
, t
0
). Let x() be the state
trajectory that results when optimal control u

is applied, starting from x


0
at time t
0
. Then
Bellmans Principle of Optimality asserts that, for any t [t
0
, t
1
], the restriction of u

to
[t, t
1
] solves P(x(t), t; x
1
, t
1
).
Exercise 2.14 Prove that Bellmans Principle of Optimality holds for the minimum energy
xed endpoint problem. [Hint: Verify that the entire derivation so far still works when |
is redened to be piecewise continuous on [t
0
, t
1
] (i.e., continuous except at nitely many
points, with nite left and right limits at those points). Then, Assuming by contradiction
existence of a lower energy control for P(x(t), t; x
1
, t
1
), show by construction that u

cannot
be optimal for P(x
0
, t
0
; x
1
, t
1
).]
It follows that, for any t such that W(t, t
1
) is invertible,
u

(t) = B
T
(t)W(t, t
1
)
1
(x(t), t) t (2.26)
which yields the closed-loop implementation, depicted in Figure 2.1, where v(t) =
B
T
(t)W
1
(t, t
1
)(t, t
1
)x
1
. In particular, if x
1
= , then v(t) = 0 t.
Exercise 2.15 Prove (2.26) from (2.25) directly, without invoking Bellmans Principle.
Example 2.2 (charging capacitor)
d
dt
cv(t) = i(t)
30 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
2.3 Fixed terminal state, unconstrained, quadratic cost
v(t)
B (t)W (t,t )
T -1
1
B(t)
A(t)
x(t)

x(t)

+
+
-
u(t)
Figure 2.1: Closed-loop implementation
i
c
r
v
+
-
Figure 2.2: Charging capacitor
minimize
t
1
_
0
ri(t)
2
dt s.t. v(0) = v
0
, v(t
1
) = v
1
We obtain
B(t)
1
c
, A(t) 0
W(0, t
1
) =
t
1
_
0
1
c
2
dt =
t
1
c
2

0
= c
2
v
0
v
1
t
1
i
0
(t) =
1
c
c
2
(v
0
v
1
)
t
1
=
c(v
1
v
0
)
t
1
= constant.
The closed-loop optimal feedback law is given by
i
0
(t) =
c
t
1
t
(v(t) v
1
).
Copyright c _19932011, Andre L. Tits. All Rights Reserved 31
Linear Optimal Control: Some (Reasonably) Simple Cases
Exercise 2.16 Discuss the same optimal control problem (with xed end points) with the
objective function replaced by
J(u)
t
1
_
t
0
u(t)
T
R(t)u(t) dt
where R(t) = R(t)
T
> 0 for all t [t
0
, t
1
] and R() is continuous. [Hint: dene a new inner
product on |.]
Remark 2.6 In view of Exercise 2.13, a control u steers (x
0
, t
0
) to (x
1
, t
1
) if and only if it
steers (x
0
(t
0
, t
1
)x
1
, t
0
) to (, t
1
). Thus there is no loss of generality in assuming x
1
= .
The controllability Gramian W(, ) happens to satisfy certain simple equations. Recalling
that
W(t, t
1
) =
t
1
_
t
(t, )B()B
T
()(t, )
T
d
one easily veries that W(t
1
, t
1
) = 0 and
d
dt
W(t, t
1
) = A(t)W(t, t
1
) + W(t, t
1
)A
T
(t) B(t)B
T
(t) (2.27)
implying that, if W(t, t
1
) is invertible, it satises
d
dt
_
W(t, t
1
)
1
_
= W(t, t
1
)
1
A(t) A
T
(t)W(t, t
1
)
1
+ W(t, t
1
)
1
B(t)B
T
(t)W(t, t
1
)
1
(2.28)
Equation (2.27) is linear. It is a Lyapunov equation. Equation (2.28) is quadratic. It is a
Riccati equation (for W(t, t
1
)
1
).
Exercise 2.17 Prove that if a matrix M(t) := W(t, t
1
) satises Lyapunov equation (2.27),
then its inverse satises Riccati equation (2.28). (Hint:
d
dt
M(t)
1
= M(t)
1
(
d
dt
M(t))M(t)
1
.)
W(, ) also satises the functional equation
W(t
0
, t
1
) = W(t
0
, t) + (t
0
, t)W(t, t
1
)
T
(t
0
, t).
As we have already seen, the Riccati equation plays a fundamental role in optimal control
systems involving linear dynamics and quadratic cost (linear-quadratic problems). At this
point, note that, if x
1
= and W(t, t
1
) is invertible, then u

(t) = B(t)
T
P(t)x(t), where
P(t) = W(t, t
1
)
1
solves Riccati equation (2.28).
We have seen that, if W(t
0
, t
1
) is invertible, the optimal cost for problem (FEP) is given
by
J(u

) = u

, u

= (x(t
0
) (t
0
, t
1
)x
1
)
T
W(t
0
, t
1
)
1
(x(t
0
) (t
0
, t
1
)x
1
). (2.29)
This is clearly true for any t
0
, so that, from a given time t < t
1
(such that W(t, t
1
) is
invertible) the cost-to-go is given by
(x(t) (t, t
1
)x
1
)
T
W(t, t
1
)
1
(x(t) (t, t
1
)x
1
).
32 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
2.3 Fixed terminal state, unconstrained, quadratic cost
This clearly bears resemblance with results we obtained for the free endpoint problem, es-
pecially when x
1
= , in which case the cost-to-go takes value x(t), W(t, t
1
)
1
x(t) where,
again, W(t, t
1
)
1
satises the associated Riccati equation.
Now consider the more general quadratic cost
J(u) :=
t
1
_
t
0
x(t),
T
L(t)x(t) + u(t)
T
u(t) dt =
t
1
_
t
0
_
x(t)
u(t)
_
T
_
L(t) 0
0 I
_ _
x(t)
u(t)
_
dt, (2.30)
where L() = L()
T
is continuous. Let K(t) = K(t)
T
be some absolutely continuous time-
dependent matrix. Using the Fundamental Lemma we see that, since x
0
and x
i
are xed, it
is equivalent to minimize

J(u) := J(u) + x
T
1
K(t
1
)x
1
x
T
0
K(t
0
)x
0
=
t
1
_
t
0
_
x(t)
u(t)
_
T
_
L(t) +

K(t) + A
T
(t)K(t) + K(t)A(t) K(t)B(t)
B(t)
T
K(t) 0
_ _
x(t)
u(t)
_
dt.
To complete the square, suppose there exists such K(t) that satises
L(t) +

K(t) + A
T
(t)K(t) + K(t)A(t) = K(t)B(t)B(t)
T
K(t)
i.e., satises the Riccati dierential equation

K(t) = A
T
(t)K(t) K(t)A(t) + K(t)B(t)B(t)
T
K(t) L(t), (2.31)
As we have seen when discussing the free endpoint problem, if L(t) is positive semi-denite
for all t, then a solution exists for every prescribed positive semi-denite nal value K(t
1
).)
Then we get

J(u) =
t
1
_
t
0
_
x(t)
u(t)
_
T
_
K(t)B(t)B(t)
T
K(t) K(t)B(t)
B(t)
T
K(t) I
_ _
x(t)
u(t)
_
dt.
=
t
1
_
t
0
_
B(t)
T
K(t)x(t) + u(t)
_
T
_
B(t)
T
K(t)x(t) + u(t)
_
dt.
Now, again supposing that some solution to (DRE) exists, let K() be any such solution
and let
v(t) = B
T
(t)K(t)x(t) + u(t).
It is readily veried that, in terms of the new control input v, the systems dynamics become
x(t) = [A(t) B(t)B
T
(t)K(t)]x(t) + Bv(t),
and the cost function takes the form

J(u) =
t
1
_
t
0
v(t)
T
v(t)dt.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 33
Linear Optimal Control: Some (Reasonably) Simple Cases
That is, we end up with the problem
minimize
t
1
_
t
0
v(t), v(t)dt
subject to x(t) = [A(t) B(t)B
T
(t)(t, K
1
, t
1
)]x(t) + Bv(t) (2.32)
x(t
0
) = x
0
x(t
1
) = x
1
where, following standard usage, we have parametrized the solutions to DRE by there value
(K
1
) at time t
1
, and denoted them by (t, K
1
, t
1
). This transformed problem (parametrized
by K
1
) is of an identical form to that we solved earlier. Denote by
ABB
T

and W
ABB
T

the state transition matrix and controllability Gramian for (2.32) (for economy of notation,
we have kept implicit the dependence of on K
1
). Then, for a given K
1
, we can write the
optimal control v
K
1
for the transformed problem as
v
K
1
(t) = B(t)
T
W
1
ABB
T

(x(t), t),
and the optimal control u

for the original problem as


u

(t) = v
K
1
(t) B(t)
T
(t, K
1
, t
1
)(x(t), t) = B(t)
T
((t, K
1
, t
1
) + W
1
ABB
T

)(x(t), t).
We also obtain, with
0
= (x
0
, t
0
),
J(u

) =

J(u

) +
T
0
(t
0
, K
1
, t
1
)
0
=
T
0
(W
ABB
T

(t
0
, t
1
)
1
+ (t
0
, K
1
, t
1
))
0
.
The cost-to-go at time t is
(x(t), t)
T
(W
ABB
T

(t, t
1
)
1
+ (t, K
1
, t
1
))(x(t), t)
Finally, if L(t) is identically zero, we can pick K(t) identically zero and we recover the
previous result.
Exercise 2.18 Show that reachability of (2.1) on [t
0
, t
1
] implies invertibility of W
ABB
T

and vice-versa.
We obtain the block diagram depicted in Figure 2.3. Thus u

(t) = B(t)
T
p(t), with
p(t) = ((t, K
1
, t
1
) + W
ABB
T

(t, t
1
)
1
)x(t).
If x
1
,= , then
w
0
(t) = B
T
(t)W
ABB
T

(t, t
1
)
1

ABB
T

(t, t
1
)x
1
.
Note that while v
0
clearly depends on K
1
, u

obviously cannot, since K


1
is an arbitrary
symmetric matrix (subject to DRE having a solution with K(t
1
) = K
1
). Thus we could have
assigned K
1
= 0 throughout the analysis. Check the details of this.
The above is a valid closed-loop implementation as it does not involve the initial point
(x
0
, t
0
) (indeed perturbations may have aected the trajectory between t
0
and the current
34 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
2.4 More general optimal control problems
time t). (t, K
1
, t
1
) can be precomputed (again, we must assume that such a solution exists.)
Also note that the optimal cost J(u

) is given by
J(uu

=

J(uu

(x
T
1
K
1
, x
1
x
T
0
(t
0
, K
1
, t
1
)x
0
)
= (x
0

ABB
T

(t
0
, t
1
)x
1
)
T
W
ABB
T

(t
0
, t
1
)
1
(x
0

ABB
T

(t
0
, t
1
)x
1
)
+ x
T
0
(t
0
, K
1
, t
1
)x
0
x
T
1
K
1
, x
1
,
and is independent of K
1
.
Finally, for all optimal control problems considered so far, the optimal control can be
expressed in terms of the adjoint variable (or co-state) p(). More precisely, the following
holds.
Theorem 2.7 Consider the xed terminal state problem with x
1
= . Suppose that the
controllability Gramian W(t
0
, t
1
) is non-singular and that the relevant Riccati equation has a
(unique) solution (t, K
1
, t
1
) on [t
0
, t
1
] with K(t
1
) = K
1
. Let x(t) be the optimal trajectory
and dene p(t) by
p(t) = ((t, K
1
, t
1
) + W
ABB
T
(t,K
1
,t
1
)
(t, t
1
)
1
)x(t)
so that the optimal control is given by
u

(t) = B
T
(t)p(t).
Prove that
_
x(t)
p(t)
_
=
_
A(t) B(t)B
T
(t)
L(t) A
T
(t)
_ _
x(t)
p(t)
_
and the optimal cost is x(t
0
)
T
p(t
0
).
Exercise 2.19 Prove the theorem.
Exercise 2.20 In all cases considered so far, the optimal control (when it exists) is in the
range space of B
T
(t) for all t. Give a simple explanation for this fact.
2.4 More general optimal control problems
We have shown how to solve optimal control problems where the dynamics are linear, the
objective function is quadratic, and the constraints are of a very simple type (xed initial
point, xed nal point). In most problems of practical interest, though, one or more of the
following features is present.
(i) nonlinear dynamics and objective function
(ii) constraints on the control or state trajectories, e.g., u(t) U t, where U R
m
(iii) more general constraints on the initial and nal state, e.g., g(x(t
1
)) 0.
To tackle such more general optimization problems, we will make use of additional mathe-
matical machinery. We rst proceed to develop such machinery.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 35
Linear Optimal Control: Some (Reasonably) Simple Cases
B(t)
A(t)

x
+
B (t)(t,K ,t )
T
1 1
A-BB
B (t)W (t,t )
T
1
-1
T
-
+
w =0
0
v
0
+
u
0
-
+
Figure 2.3: Optimal feedback law
36 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
Chapter 3
Unconstrained Optimization
References: [18, 22].
3.1 First order condition of optimality
We consider the problem
minf(x) [ x V (3.1)
where, as assumed throughout, V is a normed vector space and f : V R.
Remark 3.1 This problem is technically very similar to the problem
minf(x) : x (3.2)
where is an open set in V , as shown in the following exercise.
Exercise 3.1 Suppose V is open. Prove carefully, using the denitions given earlier,
that x is a local minimizer for (3.2) if and only if x is a local minimizer for (3.1) and x .
Now suppose f is continuously (Frechet) dierentiable (see Appendix B). (In fact, many of
the results we obtain below hold under the milder assumptions.) We next obtain a rst order
necessary condition for optimality.
Theorem 3.1 Suppose x is a local minimizer for (3.1). Then
f
x
( x) = .
Proof. Since x is a local minimizer for (3.1), there exists > 0 such that
f( x + h) f( x) h B(0, ) (3.3)
Since f is Frechet-dierentiable, we have, for all h V ,
f( x + h) = f( x) +
f
x
( x)h + o(h) (3.4)
Copyright c _19932011, Andre L. Tits. All Rights Reserved 37
Unconstrained Optimization
with
o(h)
[[h[[
0 as h 0. Hence, from (3.3), whenever h B(0, )
f
x
( x)h + o(h) 0 (3.5)
or, equivalently,
f
x
( x)(h) + o(h) 0 h B(0, ), [0, 1] (3.6)
and, dividing by , for ,= 0
f
x
( x)h +
o(h)

0 h B(0, ), (0, 1] (3.7)


It is easy to show (see exercise below) that
o(h)

0 as 0. Hence, letting 0 in
(3.7), we get
f
x
( x)h 0 h B(0, ).
Since h B(0, ) implies h B(0, ), we have
f
x
( x)(h) 0 thus
f
x
( x)h 0 h B(0, ).
Hence
f
x
( x)h = 0 h B(0, )
which implies (since
f
x
( x) is linear)
f
x
( x)h = 0 h V
i.e.,
f
x
( x) = .
Exercise 3.2 If o() is such that
o(h)
[[h[[
0 as h 0, then
o(h)

0 as 0 R.
Remark 3.2 The optimality condition above, like several other conditions derived in this
course, is only a necessary condition, i.e., a point x satisfying this condition need not be
optimal, even locally. However, if there is an optimal x, it has to be among those which satisfy
the optimality condition. Also it is clear that this optimality condition is also necessary for
a global minimizer (since a global minimizer is also a local minimizer). Hence, if a global
minimizer is known to exist, it must be, among the points satisfying the optimality condition,
the one with minimum value of f.
38 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
3.2 Steepest descent method
Exercise 3.3 Check that, if f is merely Gateaux dierentiable, the above theorem is still
valid. (Further, note that continuous dierentiability is not required.)
Suppose now that we want to nd a minimizer. Solving
f
x
( x) = for x is usually hard
and could well yield maximizers or other stationary points instead of minimizers. A central
idea is, given an initial guess x, to determine a descent, i.e., a direction in which, starting
from x, f decreases (at least for small enough displacement). Thus suppose
f
x
( x) ,= (hence
x is not a local minimizer). If h is such that
f
x
( x)h < 0 (such h exists; why?) then, for
> 0 small enough, we have

_
f
x
( x)h +
o(h)

_
< 0
and, hence, for some
0
> 0,
f( x + h) < f( x) (0,
0
].
Such h is called a descent direction for f at x. The concept of descent direction is essential
to numerical methods. If V is a Hilbert space, then
f
x
( x)h = gradf( x), h and a particular
descent direction is h = gradf( x). (This is so irrespective of which inner product (and
associated gradient) is used. We will return to this point when studying Newtons method
and variable metric methods.)
3.2 Steepest descent method
Suppose V = H, a Hilbert space. In view of what we just said, a natural algorithm for
attempting to solve (3.1) would be the following.
Algorithm 1 (steepest descent)
Data x
0
H
i = 0
while gradf(x
i
) ,= do
pick
i
arg min

f(x
i
gradf(x
i
)) : 0 (if there is no such minimizer
the algorithm fails)
x
i+1
= x
i

i
gradf(x
i
)
i = i + 1

stop
Notation: Given a real-valued function , the (possibly empty) set of global minimizers for
the problem
minimize (x) s.t. x S
is denoted by
arg min
x
(x) : x S.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 39
Unconstrained Optimization
Exercise 3.4 Let x be such that gradf(x) ,= (i.e.,
f
x
(x) ,= ). Show that

h :=

gradf(x)
gradf(x),gradf(x))
1/2
(unit vector along gradf(x)) is indeed the unique direction of (local)
steepest descent at x. First show that

h minimizes gradf(x), h subject to h, h = 1. Sec-
ond, show that
inf
h
f(x + h) [ h, h = 1 (3.8)
is approached with h close to

h for small; specically, that, given any

h ,=

h with

h,

h = 1,
there exists > 0 such that
f(x +

h) < f(x +

h) (0, ].
(Hint: Use Cauchy-Bunyakovskii-Schwartz.)
In the algorithm above, like in other algorithms we will study in this course, each iteration
consists of essentially 2 operations:
computation of a search direction (here, directly opposed to the gradient of f)
a search along that search direction, which amounts to solving, often approximately,
a minimization problem in only 1 variable, . The function () = f(x + h) can be
viewed as the one-dimensional section of f at x in direction h. This second operation
is often also called step-size computation or line search.
Before analyzing the algorithm above, we point out a practical diculty. Computation of

i
involves an exact minimization which cannot in general be performed exactly in nite
time (it requires construction of an innite sequence). Hence, point x
1
will never be actually
constructed and convergence of the sequence x
i
cannot be observed. One says that the
algorithm is not implementable, but merely conceptual. An implementable algorithm for
solving (3.1) will be examined later.
3.3 Introduction to convergence analysis
In order to analyze Algorithm 1, we embed it in a class of algorithms characterized by the
following algorithm model. Here, a : V V , V a normed vector space, represents an
iteration map; and V is a set of desirable points.
Algorithm Model 1
Data. x
0
V
i 0
while x
i
, do
x
i+1
a(x
i
)
i i + 1

stop
40 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
3.3 Introduction to convergence analysis
Theorem 3.2 Suppose there exists a function v : V R such that
(i) v() is continuous in
c
(complement of );
(ii) a() is continuous in
c
(iii) v(a(x)) < v(x) x
c
Then, if the sequence x
i
constructed by Algorithm model 1 is innite, every accumulation
point of x
i
is desirable (i.e., belongs to ).
Exercise 3.5 Prove the theorem.
Exercise 3.6 Give an example (i.e., exhibit a(), v() and ) showing that condition (ii),
in Theorem 3.2, cannot be dropped.
Remark 3.3
1. Note that Algorithm model 1 does not imply any type of optimization idea. The
result of Theorem 3.2 will hold if one can show the existence of a function v() that,
together with a(), would satisfy conditions (i) to (iii). This idea is related to that of
a Lyapunov function for the discrete system x
i+1
= a(x
i
) (but assumptions on v are
weaker, and the resulting sequence may be unbounded).
2. The result of Theorem 3.2 is stronger than it may appear at rst glance.
(i) if x
i
is bounded (e.g., if all level sets of f are bounded) and V is nite-
dimensional, accumulation points do exist.
(ii) in such case, if is a nite set (which it is in lots of practical cases), there are
simple techniques (such as step-size limitation) that can force the entire sequence
to converge (to an element of ).
Note on the other hand that it cannot possibly be guaranteed that x
i
x for some
x . Indeed, for one thing, may be empty.
Exercise 3.7 Consider Algorithm 1 (steepest descent) with H = R
n
and dene, for any
x R
n
(x) = arg min

f(x f(x)) : 0
where we have to assume that (x) is uniquely dened (unique global minimizer) for every
x R
n
. Also suppose that () is locally bounded, i.e., for any bounded set K, there exists
M > 0 s.t. [(x)[ < M for all x K. Show that the hypotheses of Theorem 3.2 are satised
with = x R
n
:
f
x
(x) = 0 and v = f. [Hint: the key point is to show that a()
is continuous, i.e., that () is continuous. This does hold because, since f is continuously
dierentiable, the curve below (Figure 3.1) does not change too much in a neighborhood of
x.]
Copyright c _19932011, Andre L. Tits. All Rights Reserved 41
Unconstrained Optimization
0
(x)

f(x-f(x))-f(x)

Figure 3.1:
Remark 3.4 We just proved that Algorithm 1 yields accumulation points x (if any) such
that
f
x
( x) = 0. There is no guarantee, however, that x is even a local minimizer (e.g., take
the case where x
0
is a local maximizer). Nevertheless, this will very likely be the case, since
the cost function decreases at each iteration (and thus, local minimizers are the only stable
points.)
In many cases (x) will not be uniquely dened for all x and hence a() will not be continuous.
We will now consider a more general algorithm model involving a point-to-set iteration map
A : V 2
V
(2
V
is the set of subsets of V ).
Algorithm Model 2
Data. x
0
V
i 0
while x
i
, do
pick x
i+1
A(x
i
)
i i + 1

stop
The advantages of using a point to set iteration map are that
(i) compound algorithms can be readily analyzed (two or more algorithms are intertwined)
(ii) this can include algorithms for which the iteration depends on some past information
(i.e., conjugate gradient methods)
(iii) algorithms not satisfying the conditions of the previous theorem (e.g., a not continuous)
may satisfy the conditions of the theorem below.
The algorithm above, with the convergence theorem below, will allow us to analyze an
implementable algorithm. The following theorem is due to Polak [22].
Theorem 3.3 Suppose that there exists a function v : V R such that
(i) v() is continuous in
c
(ii) x
c
> 0, > 0 such that
42 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
3.3 Introduction to convergence analysis
v(y

) v(x

) x

B(x, ) y

A(x

) (3.9)
Then, if the sequence x
i
constructed by Algorithm model 2 is innite, every accumulation
point of x
i
is desirable (i.e., belongs to ).
Remark 3.5 (3.9) indicates a uniform decrease in the neighborhood of any non-desirable
point. Note that a similar property is implied by (ii) and (iii) in the previous theorem.
Lemma 3.1 Let
i
R be a monotonically decreasing sequence such that
i
K

for
some K IN,

R. Then
i

.
Exercise 3.8 Prove the lemma.
Proof of Theorem 3.3
By contradiction. Suppose x
i
K
x , . Since v() is continuous, v(x
i
)
K
v( x). Since, in
view of (ii),
v(x
i+1
) < v(x
i
) i
it follows from the lemma above that
v(x
i
) v( x) . (3.10)
Now let , correspond to x in assumption (ii). Since x
i
K
x, there exists i
0
such that
i i
0
, i K, x
i
belongs to B( x, ). Hence, i i
0
, i K,
v(y) v(x
i
) y A(x
i
) (3.11)
and, in particular
v(x
i+1
) v(x
i
) i i
0
, i K (3.12)
But this contradicts (3.10) and the proof is complete.
Exercise 3.9 Show that Algorithm 1 (steepest descent) with H = R
n
satises the assump-
tions of Theorem 3.3. Hence x
i
K
x implies
f
x
( x) = 0 (assuming that argmin

f(x
i

f(x
i
)) is always nonempty).
In the following algorithm, a line search due to Armijo replaces the exact line search of Algo-
rithm 1, making the algorithm implementable. This line search imposes a decrease of f(x
i
)
at each iteration, which is common practice. Note however that such monotone decrease
in itself is not sucient for inducing convergence to stationary points. Two ingredients in
the Armijo line search insure that sucient decrease is achieved: (i) the back-tracking
technique insures that, away from stationary points, the step will not be vanishingly small,
and (ii) the Armijo line test insures that, whenever a reasonably large step is taken, a
reasonably large decrease is achieved.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 43
Unconstrained Optimization
We present this algorithm in a more general form, with a search direction h
i
(h
i
=
gradf(x
i
) corresponds to Armijo-gradient).
Algorithm 2 (Armijo step-size rule)
Parameters , (0, 1)
Data x
0
V
i = 0
while
f
x
(x
i
) ,= do
compute h
i
according to some scheme
:= 1
while f(x
i
+ h
i
) f(x
i
) >
f
x
(x
i
)h
i
do
:=
x
i+1
:= x
i
+ h
i
i := i + 1

stop
To get an intuitive picture of this step-size rule, let us dene a function
i
: R R by

i
() = f(x
i
+ h
i
) f(x
i
)
Using the chain rule we have

i
(0) =
f
x
(x
i
+ 0h
i
)h
i
so that the condition to be satised by can be written as

i
(
k
)

i
(0)
Hence the Armijo rules prescribes to choose the step-size
i
as the smallest power of (hence
the largest number since < 1) at which the curve () is below the straight line

i
(0),
as shown on the Figure 3.2. In the case of the gure k = 2 will be chosen. We see that this
step-size will be well dened as long as
f
x
(x
i
)h
i
< 0
which insures that the straight lines on the picture are downward. Let us state and prove
this precisely.
Proposition 3.1 Suppose that
f
x
(x
i
) ,= 0 and that h
i
is such that
f
x
(x
i
)h
i
< 0
Then there exists an integer k such that =
k
satises let line search criterion.
44 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
3.3 Introduction to convergence analysis
Proof
() = () (0) =

(0) + o()
=

(0) + (1 )

(0) + o()
=

(0) + (1 )

(0) +
o()

,= 0
Since

(0) =
f
x
(x
i
)h
i
< 0, the expression within braces is negative for > 0 small enough,
thus () <

(0) for > 0 small enough.


We will now apply Theorem 3.3 to prove convergence of Algorithm 2. We just have to show
that condition (ii) holds (using v f, (i) holds by assumption).
Theorem 3.4 Suppose V is a Hilbert space. Let H(x) denote the set of search directions
that could possibly be constructed by Algorithm 2 when x is the current iterate (in the case
of steepest descent, H(x) = gradf(x)). Suppose that H(x) is bounded away from zero
near non-stationary points, i.e., for any x such that
f
x
( x) ,= 0, there exists > 0 such that
inf|h| : h H(x), |x x| > 0. (3.13)
Further suppose that for any x for which
f
x
( x) ,= 0 there exist positive numbers and such
that, x B( x, ), h H(x), it holds
gradf(x), h [[gradf(x)[[ [[h[[, (3.14)
where || is the norm induced by underlying inner product. Then x
i
K
x

implies
f
x
(x

) = 0.
Proof (For simplicity we assume the standard Euclidean inner product.)
f(x + h) = f(x) +
_
1
0
f(x + th), hdt
0

i
(0)=f(x ),h
i i

1
1=
0
(0)=f(x ),h
i i i
()=f(x +h )-f(x )
i i i i
Figure 3.2:
Copyright c _19932011, Andre L. Tits. All Rights Reserved 45
Unconstrained Optimization
Thus
f(x + h) f(x) f(x), h =
_
1
0
f(x + th) f(x), hdt
+ (1 )f(x), h (3.15)
( sup
t[0,1]
[f(x + th) f(x), h[ + (1 )f(x), h) 0
Suppose now that x is such that f( x) ,= 0 and let , and C satisfy the hypotheses of the
theorem. Substituting (3.14) into (3.15) yields (since (0, 1) and 0), using Schwartz
inequality,
f(x + h) f(x) f(x), h [[h[[( sup
t[0,1]
[[f(x + th) f(x)[[ (1 )[[f(x)[[)
x B( x, ), h H( x) (3.16)
Assume now that h(x) = f(x) [the proof for the general case is left as a (not entirely
trivial) exercise.] First let us pick

(0, ] s.t., for some > 0, (using continuity of f)


(1 )[[f(x)[[ > > 0 x B( x,

)
Also by continuity of f, C s.t. [[f(x)[[ C x B( x,

). Since

B( x,

) is compact,
f is uniformly continuous over

B( x,

). Thus, there exists > 0 such that


[[f(x + v) f(x)[[ < [[v[[ < x

B( x,

)
Thus,
[[f(x tf(x)) f(x)[[ < t [0, 1], [0,

C
], x

B( x,

)
which implies
sup
t[0,1]
[[f(x tf(x)) f(x)[[ < [0,

], x

B( x,

)
with

=

C
> 0. Thus (3.16) yields
f(x f(x)) f(x) + [[f(x)[[
2
< 0 (0,

], x

B( x,

) (3.17)
Let us denote by k(x) the value of k
i
constructed by Algorithm 2 if x
i
= x. Then, from
(3.17) and the denition of k(x)
k(x) k


= max(0,

k) x B( x,

) (3.18)
where

k is such that


<

k1
(3.19)
(since

k
will then always satisfy inequality (3.17). The iteration map A(x) (singleton valued
in this case) is
A(x) = x
k(x)
f(x).
46 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
3.3 Introduction to convergence analysis
and, from the line search criterion, using (3.18) and (3.19)
f(A(x)) f(x)
k(x)
[[f(x)[[
2

k

[[f(x)[[
2
x

B( x,

)

k

2
(1 )
2

2
x

B( x,

)
and condition (ii) of Theorem 3.3 is satised with
=
k

2
(1 )
2

2
> 0.
Hence x
i
K
x

implies x

, i.e., f(x

) = 0.
Remark 3.6 Condition (3.14) expresses that the angle between h and (gradf(x)), (in the
2D plane spanned by these two vectors) is uniformly bounded by cos
1
() (note that > 1
cannot possibly satisfy (3.14) except if both sides are = 0). In other words, this angle is
uniformly bounded away from 90

. This angle just being less than 90

for all x, insuring that


h(x) is always a descent direction, is indeed not enough. Condition (3.13) prevents h(x)
from collapsing (resulting in a very small step [[x
k+1
x
k
[[) except near a desirable point.
Exercise 3.10 Prove that Theorem 3.4 is still true if the Armijo search is replaced by an
exact search (as in Algorithm 1).
Remark 3.7
1. A key condition in Theorem 3.3 is that of uniform descent in the neighborhood of a
non-desirable point. Descent by itself may not be enough.
2. The best values for and in Armijo step-size are not obvious a priori. More about
all this can be found in [22, 4].
3. Many other step-size rules can be used, such as golden section search, quadratic or cubic
interpolation, Goldstein step-size. With some line searches, a stronger convergence
result than that obtained above can be proved, under the additional assumption that f
is bounded from below (note that, without such assumption, the optimization problem
would not be well dened): Irrespective of whether or not x
k
has accumulation
points, the sequence of gradients f(x
k
) always converges to zero. See, e.g., [20,
section 3.2].
Exercise 3.11 Let f : R
n
R be continuously dierentiable. Suppose x
i
is constructed
by the Armijo-steepest-descent algorithm. Suppose that x :
f
x
(x) = 0 is nite and suppose
that x
i
has an accumulation point x. Then x
i
x.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 47
Unconstrained Optimization
Exercise 3.12 Let x be an isolated stationary point (i.e., > 0 s.t. x ,= x x B( x, ),
f
x
(x) ,= 0) and let x
i
be a sequence such that all its accumulation points are stationary.
Suppose that x
i
K
x for some K and that there exists a neighborhood N( x) of x such that
[[x
i+1
x
i
[[ 0 as i , x
i
N( x). Then x
i
x.
The assumption in Exercise 3.12 is often satised. In particular, any local minimum satisfying
the 2nd order sucient condition of optimality is isolated (why?). Finally if x
i+1
= x
i

gradf(x
i
) and [[ 1 (e.g., Armijo-gradient) then |x
i+1
x
i
| |gradf(x
i
)| which goes to
0 on sub-sequences converging to a stationary point; for other algorithms, suitable step-size
limitation schemes will yield the same result.
Exercise 3.13 Let f : R
n
R be continuously dierentiable and let x
i
be a bounded
sequence with the property that every accumulation point x satises
f
x
( x) = 0. Then
f
x
(x
i
) 0 as i .
3.4 Minimization of convex functions
Convex functions have very nice properties in relation with optimization as shown by the
theorem below.
Exercise 3.14 The set of global minimizers of a convex function is convex (without any
dierentiability assumption). Also, every local minimizer is global. If such minimizer exists
and f is strictly convex, then it is the unique global minimizer.
Theorem 3.5 Suppose f : V R, is continuously dierentiable and convex. Then
f
x
(x

) = 0 implies that x

is a global minimizer for f. Furthermore, if f is strictly convex,


x

is strict and is the unique global minimizer.


Proof. If f is convex, then, x V
f(x) f(x

) +
f
x
(x

)(x x

)
and, since
f
x
(x

) = 0
f(x) f(x

) x V
and x

is a global minimizer. If f is strictly convex, then x ,= x

f(x) > f(x

) +
f
x
(x

)(x x

)
hence
f(x) > f(x

) x ,= x

and x

is strict and is the unique global minimizer.


48 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
3.5 Second order optimality conditions
Remark 3.8 Let V = R
n
and suppose f is strongly convex. Then there is a global mini-
mizer (why?). By the previous theorem it is unique and
f
x
vanishes at no other point.
Exercise 3.15 Suppose that f is strongly convex and that the sequence x
i
is such that
(i) any accumulation point x (if there is one) satises
f
x
( x) = 0 (so that x is the unique
global minimizer.)
(ii) i, f(x
i+1
) f(x
i
)
Then x
i
x.
3.5 Second order optimality conditions
Here we consider only V = R
n
(the analysis in the general case is slightly more involved).
Consider again the problem
minf(x) [ x R
n
(3.20)
Theorem 3.6 (2nd order necessary condition) Suppose that f is twice dierentiable and let
x be a local minimizer for (3.20). Then
2
f( x) is positive semi-denite, i.e.
d
T

2
f( x)( x)d 0 d R
n
Proof. Let d R
n
0 and t > 0. Since x is a local minimizer, f( x) = 0. Second order
expansion of f around x yields
0 f( x + td) f( x) =
t
2
2
_
d
T

2
f( x)d +
o
2
(td)
t
2
_
(3.21)
with
o
2
(h)
|h|
2
0 as h 0. The claim then follows by letter t 0.
Theorem 3.7 (2nd order suciency condition) Suppose that f is twice dierentiable, that
f
x
( x) = 0 and that
2
f( x) is positive denite. Then x is a strict local minimizer for (3.20).
Proof. Let m > 0 be the smallest eigenvalue of
2
f( x). (It is positive indeed since V is
assumed nite-dimensional.) Then
f( x + h) f( x) |h|
2
_
m
2
+
o
2
(h)
|h|
2
_
h ,= 0 .
Let > 0 be such that
[o
2
(h)[
|h|
2
<
m
2
for all |h| < . Then
f( x + h) > f( x) |h| ,
proving the claim.
Alternatively, under the further assumption that the second derivatives of f is continuous,
Theorem 3.7 can be proved by making use of (B.15), and using the fact that, due to the
assumed continuity of the second derivative,
_

2
f
x
2
( x + th)h
_
h (m/2)|h|
2
for all h small
enough and t (0, 1).
Copyright c _19932011, Andre L. Tits. All Rights Reserved 49
Unconstrained Optimization
Exercise 3.16 Show by a counterexample that the 2
nd
order suciency condition given
above is not valid when the space is innite-dimensional. [Hint: Consider the space of
sequences in R, with an appropriate norm.] Show that the condition remains sucient in
the innite-dimensional case if it is expressed as: there exists m > 0 such that, for all h,
_

2
f
x
2
( x)h
_
h (m/2)|h|
2
Remark 3.9 Consider the following proof for (2). Let d ,= 0. Then d,

2
f
x
2
( x)d = > 0.
Let h = td. Proceeding as in (1), (3.21) yields f( x + td) f( x) > 0 t (0,

t] for some

t > 0, which shows that x is a local minimizer. This argument is in error. Why? (Note that
if this argument was correct, it would imply that the result also holds on innite-dimensional
spaces.)
Remark 3.10 Note that, in the proof above, it is not enough that o
2
(h)/|h|
2
goes to zero
along straight lines. (Thus twice Gateaux dierentiable is not enough.)
Exercise 3.17 Exhibit an example where f has a strict minimum at some x, with

2
f
x
2
( x) _ 0
(as required by the 2nd order necessary condition), but such that there is no neighborhood of
x where

2
f
x
2
(x) is everywhere positive semi-denite. (Try x R
2
; while examples in R do
exist, they are contrived.) Simple examples exist where there is a neighborhood of x where
the Hessian is nowhere positive semi-denite (except at x). [Hint: First ignore the strictness
requirement.]
3.6 Conjugate direction methods
(see [18])
We restrict the discussion to V = R
n
.
Steepest descent type methods can be very slow.
Exercise 3.18 Let f(x, y) =
1
2
(x
2
+ ay
2
) where a > 0. Consider the steepest descent al-
gorithm with exact minimization. Given (x
i
, y
i
) R
2
, obtain formulas for x
i+1
and y
i+1
.
Using these formulas, give a qualitative discussion of the performance of the algorithm for
a = 1, a very large and a very small. Verify numerically using, e.g., MATLAB.
If the objective function is quadratic and x R
2
, two function evaluations and two gradient
evaluations are enough to identify the function exactly (why?). Thus there ought to be a way
to reach the solution in two iterations. As most functions look quadratic locally, such method
should give good results in the general case. Clearly, such a method must have memory (to
remember previous function and gradient values). It turns out that a very simple idea gives
answers to these questions. The idea is that of conjugate direction.
Denition 3.1 Given a symmetric matrix Q, two vectors d
1
and d
2
are said to be Q-
orthogonal, or conjugate with respect to Q if d
T
1
Qd
2
= 0
50 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
3.6 Conjugate direction methods
Fact. If Q is positive denite and d
0
, . . . , d
k
are Q-orthogonal and are all nonzero, then
these vectors are linearly independent (and thus there can be no more than n such vectors).
Proof. See [18].
Theorem 3.8 (Expanding Subspace Theorem)
Consider the quadratic function f(x) =
1
2
x
T
Qx +b
T
x with Q 0 and let h
0
, h
1
, . . . , h
n1
be
a sequence of Q-orthogonal vectors in R
n
. Then given any x
0
R
n
if the sequence x
k
is
generated according to x
k+1
= x
k
+
k
h
k
, where
k
minimizes f(x
k
+h
k
), then x
k
minimizes
f over the ane set x
0
+ span h
0
, . . . , h
k1
.
Proof. Since f is convex we can write
f(x
k
+ h) f(x
k
) +f(x
k
)
T
h
Thus it is enough to show that, for all h sp h
0
, . . . , h
k1
,
f(x
k
)
T
h = 0
i.e., since h span h
0
, . . . , h
k1
if and only if h span h
0
, . . . , h
k1
, that
f(x
k
)
T
h = 0 h span h
0
, . . . , h
k1
,
which holds if and only if
f(x
k
)
T
h
i
= 0 i = 0, . . . , k 1.
We prove this by induction on k, for i = 0, 1, 2, . . . First, for any i, and k = i + 1,
f(x
i+1
)
T
h
i
=

f(x
i
+ h
i
) = 0
Suppose it holds for some k > i. Then
f(x
k+1
)
T
h
i
= (Qx
k+1
+ b)
T
h
i
= (Qx
k
+
k
Qh
k
+ b)
T
h
i
= f(x
k
)
T
h
i
+
k
Qh
T
k
h
i
= 0
This rst term vanishes due to the induction hypothesis, the second due to Q-orthogonality
of the h

i
s.
Corollary 3.1 x
n
minimizes f(x) =
1
2
x
T
Qx + b
T
x over R
n
, i.e., the given iteration yields
the minimizer for any quadratic function in no more than n iterations.
Remark 3.11 The minimizing step size
k
is given by
Copyright c _19932011, Andre L. Tits. All Rights Reserved 51
Unconstrained Optimization

k
=
(Qx
k
+ b)
T
h
k
h
T
k
Qh
k
Conjugate gradient method
There are many ways to choose a set of conjugate directions. The conjugate gradient method
selects each direction as the negative gradient added to a linear combination of the previous
directions. It turns out that, in order to achieve Q-orthogonality, one must use
h
k+1
= f(x
k+1
) +
k
h
k
(3.22)
i.e., only the preceding direction h
k
has a nonzero coecient. This is because, if (3.22) was
used to construct the previous iterations, then f(x
k+1
) is already conjugate to h
0
, . . . , h
k1
.
Indeed, rst notice that h
i
is always a descent direction (unless f(x
i
) = 0), so that
i
,= 0.
Then, for i < k
f(x
k+1
)
T
Qh
i
= f(x
k+1
)
T
_
1

i
Q(x
i+1
x
i
)
_
=
1

i
f(x
k+1
)
T
(f(x
i+1
) f(x
i
))
=
1

i
f(x
k+1
)
T
(
i
h
i
h
i+1
+ h
i

i1
h
i1
) = 0
where we have used (3.22) for k = i + 1 and k = i, and the Expanding Subspace Theorem.
The coecient
k
is chosen so that h
T
k+1
Qh
k
= 0. One gets

k
=
f(x
k+1
)
T
Qh
k
h
T
k
Qh
k
. (3.23)
Non-quadratic objective functions
If x

is a minimizer for f,
2
f(x

) is positive semi-denite. Generically (i.e., in most cases),


it will be strictly positive denite, since matrices are generically non-singular. Thus (since
f(x

) = 0)
f(x) = f(x

) +
1
2
(x x

)
T
Q(x x

) + o
2
(x x

)
with Q =
2
f(x

) 0. Thus, close to x

, f looks like a quadratic function with positive


denite Hessian matrix and the conjugate gradient algorithm should work very well in such
a neighborhood of the solution. However, (3.23) cannot be used for
k
since Q is unknown
and since we do not want to compute the second derivative. Yet, for a quadratic function,
it can be shown that

k
=
[[f(x
k+1
[[
2
[[f(x
k
)[[
2
=
(f(x
k+1
) f(x
k
))
T
f(x
k+1
)
[[f(x
k
)[[
2
(3.24)
The rst expression yields the Fletcher-Reeves conjugate gradient method. The second one
gives the Polak-Ribi`ere conjugate gradient method.
52 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
3.6 Conjugate direction methods
Exercise 3.19 Prove (3.24).
Algorithm 3 (conjugate gradient, Polak-Ribi`ere version)
Data x
o
R
n
i = 0
h
0
= f(x
0
)
while f(x
i
) ,= 0 do

i
arg min

f(x
i
+ h
i
) (exact search)
x
i+1
= x
i
+
i
h
i
h
i+1
= f(x
i+1
) +
i
h
i
i = i + 1

stop
The Polak-Ribi`ere formula uses

i
=
(f(x
i+1
) f(x
i
))
T
f(x
i+1
)
[[f(x
i
)[[
2
. (3.25)
It has the following advantage over the Fletcher-Reeves formula: away from a solution there
is a possibility that the search direction obtained be not very good, yielding a small step
[[x
k+1
x
k
[[. If such a diculty occurs, [[f(x
k+1
) f(x
k
)[[ will be small as well and P-R
will yield f(x
k+1
) as next direction, thus resetting the method.
By inspection, one veries that h
i
is a descent direction for f at x
i
, i.e., f(x
i
)
T
h
i
< 0
whenever f(x
i
) ,= 0. The following stronger statement can be proved.
Fact. If f is twice continuously dierentiable and strongly convex then > 0 such that
f(x
i
), h
i
[[f(x
i
)[[ [[h
i
[[ i
where h
i
and x
i
are as constructed by Algorithm 3 (in particular, this assumes an exact
line search).
Exercise 3.20 Show that
[[h
i
[[ [[f(x
i
)[[ i.
As pointed out earlier one can show that the convergence theorem of Algorithm 2 still holds
in the case of an exact search (since an exact search results in a larger decrease).
Exercise 3.21 Show how the theorem just mentioned can be applied to Algorithm 3, by
specifying H(x).
Thus, all accumulation points are stationary and, since f is strongly convex, x
i
x the
unique global minimizer of f. An implementable version of Algorithm 3 can be found in [22].
If f is not convex, it is advisable to periodically reset the search direction (i.e., set h
i
=
f(x
i
) whenever i is a multiple of some number k; e.g., k = n to take advantage of the
quadratic termination property).
Copyright c _19932011, Andre L. Tits. All Rights Reserved 53
Unconstrained Optimization
3.7 Rates of convergence
(see [21])
Note: Our denitions are simpler and not exactly equivalent to the ones in [21].
Quotient convergence rates
Denition 3.2 Suppose x
i
x

. One says that x


i
converges to x

with a Q-order of
p( 1) and a corresponding Q-factor = if there exists i
0
such that, for all i i
0
[[x
i+1
x

[[ [[x
i
x

[[
p
.
Obviously, for a given initial point x
0
, and supposing i
0
= 0, the larger the Q-order, the faster
x
i
converges and, for a given Q-order, the smaller the Q-factor, the faster x
i
converges.
Also, according to our denition, if x
i
converges with Q-order = p it also converges with
Q-order = p

for any p

p, and if it converges with Q-order = p and Q-factor = it


also converges with Q-order = p and Q-factor =

for any

. Thus it may be more


appropriate to say that x
i
converges with at least Q-order = p and at most Q-factor =
. But what is more striking is the fact that a larger Q-order will overcome any initial
conditions, as shown in the following exercise. (If p = 1, a smaller Q-factor also overcomes
any initial conditions.)
Exercise 3.22 Let x
i
x

be such that [[x


i+1
x

[[ = [[x
i
x

[[
p
for all i, with > 0,
p 1, and suppose that y
i
y

is such that
[[y
i+1
y

[[ [[y
i
y

[[
q
i for some > 0
Show that, if q > p, for any x
0
, y
0
, x
0
,= x

N such that
[[y
i
y

[[ < [[x
i
x

[[ i N
Q-linear convergence
If (0, 1) and p = 1, convergence is called Q-linear. This terminology comes from the fact
that, in that case, we have (assuming i
0
= 0),
[[x
i
x

[[
i
[[x
0
x

[[ i
and, hence
log [[x
i
x

[[ i log + log [[x


0
x

[[ i
so that log |x
i
x

| (which, when positive, is roughly proportional to the number of exact


gures in x
i
) is linear (more precisely, ane) in i.
If x
i
x

Q-linearly with Q-factor= , clearly


[[x
i+1
x

[[
[[x
i
x

[[
for i larger enough. This
motivates the following denition.
54 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
3.7 Rates of convergence
Denition 3.3 x
i
x

Q-superlinearly if
[[x
i+1
x

[[
[[x
i
x

[[
0 as i
Exercise 3.23 Show, by a counterexample, that a sequence x
i
can converge Q-superlinearly,
without converging with any Q-order p > 1. (However, Q-order larger than 1 implies Q-
superlinear convergence.)
Exercise 3.24 Show that, if x
i
x

Q-superlinearly, [[x
i+1
x
i
[[ is a good estimate of the
current error [[x
i
x

[[ in the sense that


lim
i
[[x
i
x

[[
[[x
i+1
x
i
[[
= 1
Denition 3.4 If x
i
x

with a Q-order p = 2, x
i
is said to converge Q-quadratically.
Q-quadratic convergence is very fast: for large i, the number of exact gures is doubled at
each iteration.
Exercise 3.25 [21]. The Q-factor is norm-dependent (unlike the Q-order). Let
x
i
_
_
_
(.9)
k
_
1
0
_
for k even
(.9)
k
_
1/

2
1/

2
_
for k odd
and consider the norms
_
x
2
1
+ x
2
2
and max([x
1
[, [x
2
[). Show that in both cases the sequence
converges with Q-order= 1, but that only in the 1st case convergence is Q-linear ( > 1 in
the second case).
Root convergence rates
Denition 3.5 One says that x
i
x

with an R-order equal to p 1 and an R-factor equal


to (0, 1) if there exists i
0
such that, for all i i
0
[[x
i
0
+i
x

[[
i
for p = 1 (3.26)
[[x
i
0
+i
x

[[
p
i
for p > 1 (3.27)
for some > 0. (Note: by increasing , i
0
can always be set to 0.)
Equivalently there exists

> 0 and

(0, 1) such that, for all i,


[[x
i
x

[[
i

for p = 1 (3.28)
[[x
i
x

[[
p
i

p > 1 (3.29)
(take

=
1/p
i
0
).
Again with the denition as given, it would be appropriate to used the phrases at least and
at most.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 55
Unconstrained Optimization
Exercise 3.26 Show that, if x
i
x

with Q-order= p 1 and Q-factor= (0, 1) then


x
i
x

with R-order= p and R-factor= . Show that the converse is not true. Finally,
exhibit a sequence x
i
converging
(i) with Q-order p 1 which does not converge with any R-order= p

> p, with any


R-factor
(ii) with Q-order p 1 and Q-factor , not converging with R-order p with any R-factor

< .
Denition 3.6 If p = 1 and (0, 1), convergence is called R-linear.
If x
i
x

R-linearly we see from (3.28) that


limsup
i
[[x
i
x

[[
1
i
lim
i

1
i
=
Denition 3.7 x
i
is said to converge to x

R-superlinearly if
lim
i
[[x
i
x

[[
1
i
= 0
Exercise 3.27 Show that, if x
i
x

Q-superlinearly, then x
i
x

R-superlinearly.
Exercise 3.28 Show, by a counterexample, that a sequence x
i
can converge R-superlinearly
without converging with any R-order p > 1
Denition 3.8 If x
i
x

with R-order p = 2, x
i
is said to converge R-quadratically.
Remark 3.12 R-order, as well as R-factor, are norm independent.
Rate of convergence of rst order algorithms (see [22, 18])
Consider the following algorithm
Algorithm
Data x
0
R
n
i = 0
while
f
x
(x
i
) ,= 0 do
obtain h
i

i
arg minf(x
i
+ h
i
) [ 0
x
i+1
= x
i
+
i
h
i
i = i + 1

stop
Suppose that there exists > 0 such that
f(x
i
)
T
h
i
[[f(x
i
)[[ [[h
i
[[ i, (3.30)
where | | is the Euclidean norm.
56 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
3.7 Rates of convergence
and that h
i
,= 0 whenever
f
x
(x
i
) ,= 0 (condition [[h
i
[[ c[[
f
x
(x
i
)[[ is obviously superuous
if we use an exact search, since, then, x
i+1
depends only on the direction of h
i
). We know
that this implies that any accumulation point is stationary. We now suppose that, actually,
x
i
x

. This will happen for sure if f is strongly convex. It will in fact happen in most
cases when x
i
has some accumulation point.
Then, we can study the rate of convergence of x
i
. We give the following theorem without
proof (see [22]).
Theorem 3.9 Suppose x
i
x

, for some x

, where x
i
is constructed by the algorithm
above, and assume that (3.30) holds. Suppose that f is twice continuously dierentiable and
that the second order suciency condition holds at x

(as discussed above, this is a mild


assumption). Let m, M, be positive numbers such that x B(x

, ), y R
n
m[[y[[
2
y
T

2
f(x)y M[[y[[
2
[Such numbers always exist. Why?]. Then x
i
x

R-linearly (at least) with an R-factor of


(at most)
=
_
1 (
m
M
)
2
(3.31)
If = 1 (steepest descent), convergence is Q-linear with same Q-factor (with Euclidean
norm).
Exercise 3.29 Show that if < 1 convergence is not necessarily Q-linear (with Euclidean
norm).
Remark 3.13
1. f as above could be called locally strongly convex, why?
2. without knowing anything else on h
i
, the fastest convergence is achieved with = 1
(steepest descent), which yields
s
=
_
1 (
m
M
)
2
. m and M are related to the smallest
and largest eigenvalues of

2
f
x
2
(x

) (why? how?). If m M, convergence may be very


slow again (see Figure 3.3) (also see [4]).
m << M
m M
level curves

x
2
x
0
x
1
x
0
x x
1
*
Figure 3.3:
Copyright c _19932011, Andre L. Tits. All Rights Reserved 57
Unconstrained Optimization
We will see below that if h
i
is cleverly chosen (e.g., conjugate gradient method) the rate of
convergence can be much faster.
If instead of using exact line search we use an Armijo line search, with parameters ,
(0, 1), convergence is still R-linear and the R-factor is now given by

a
=
_
1 4(1 )(
m
M
)
2
(3.32)
Remark 3.14
1. For = 1/2 and 1,
a
is close to
s
, i.e., Armijo gradient converges as fast as
steepest descent. Note, however that, the larger is, the more computer time is going
to be needed to perform the Armijo line search, since
k
will be very slowly decreasing
when k increases.
2. Hence it appears that the rate of convergence does not by itself tell how fast the
problem is going to be solved (even asymptotically). The time needed to complete
one iteration has to be taken into account. In particular, for the rate of convergence
to have any signicance at all, the work per iteration must be bounded. See exercise
below.
Exercise 3.30 Suppose x
i
x

with x
i
,= x

for all i. Show that for all p, there exists a


sub-sequence x
i
k
such that x
i
k
x

with Q-order = p and Q-factor = .


Exercise 3.31 Consider two algorithms for the solution of some given problem. Algo-
rithms 1 and 2 construct sequences x
1
k
and x
2
k
, respectively, both of which converge to
x

. Suppose x
1
0
= x
2
0
B(x

, 1) (open unit ball) and suppose


|x
i
k+1
x

| = |x
i
k
x

|
p
i
with p
1
> p
2
> 0. Finally, suppose that, for both algorithms, the CPU time needed to generate
x
k+1
from x
k
is bounded (as a function of k), as well as bounded away from 0. Show that
there exists > 0 such that, for all (0, ), x
1
k
enters the ball B(x

, ) in less total
CPU time than x
2
k
does. Thus, under bounded, and bounded away from zero, time per
iteration, Q-orders can be meaningfully compared. (This is in contrast with the point made
in Exercise 3.30.)
Note on the assumption x
i
x

. The convergence results given earlier assert that,


under some assumptions, every accumulation point of the sequence generated by a suitable
algorithm (e.g., Armijo gradient) satises the rst order necessary condition of optimality.
A much nicer result would be that the entire sequence converges. The following exercise
addresses this question.
Conjugate direction methods
58 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
3.7 Rates of convergence
The rates of convergence (3.31) and (3.32) are conservative since they do not take into
account the way the directions h
i
are constructed, but only assume that they satisfy (3.30).
The following modication of Algorithm 3 can be shown to converge superlinearly.
Algorithm 4 (conjugate gradient with periodic reinitialization)
Parameter k > 0 (integer)
Data x
0
R
n
i = 0
h
0
= f(x
0
)
while f(x
i
) ,= 0 do
pick
i
argminf(x
i
+ h
i
) : 0
x
i+1
= x
i
+
i
h
i
h
i+1
= f(x
i+1
) +
i
h
i
with
i
=
_
0 if i is a multiple of k
(f(x
i+1
)f(x
i
))
T
f(x
i+1
)
[[f(x
i
)[[
2
otherwise
i = i + 1

stop
Exercise 3.32 Show that any accumulation point of the sub-sequence x
i

i=k
of the se-
quence x
i
constructed by Algorithm 4 is stationary.
Exercise 3.33 (Pacer step) Show that, if f is strongly convex in any bounded set, then
the sequence x
i
constructed by Algorithm 4 converges to the minimum x

and the rate of


convergence is at least R-linear.
Convergence is in fact n-step Q-quadratic if k = n. If it is known that x
i
x

, clearly the
strong convexity assumption can be replaced by strong convexity around x

, i.e., 2nd order


suciency condition.
Theorem 3.10 Suppose that k = n and suppose that the sequence x
i
constructed by
Algorithm 4 is such that x
i
x

, at which point the 2nd order suciency condition is


satised. Then x
i
x

n-step Q-quadratically, i.e. q, l


0
such that
[[x
i+n
x

[[ q[[x
i
x

[[
2
for i = ln, l l
0
.
This should be compared with the quadratic rate obtained below for Newtons method:
Newtons method achieves the minimum of a quadratic convex function in 1 step (compared
to n steps here).
Exercise 3.34 Show that n-step Q-quadratic convergence does not imply R-superlinear con-
vergence. Show that the implication would hold under the further assumption that, for some
C > 0,
|x
k+1
x

| C|x
k
x

| k.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 59
Unconstrained Optimization
3.8 Newtons method
Let us rst consider Newtons method for solving a system of equations. Consider the system
of equations
F(x) = 0 (3.33)
with F : V V , V a normed vector space, and F is dierentiable. The idea is to replace
(3.33) by a linear approximation at the current estimate of the solution (see Figure 3.4).
Suppose x
i
is the current estimate. Assuming Frechet-dierentiability, consider the equation
F(x)
convergent
x
1
x
0
x
2
x
n=1

Figure 3.4:

F
i
(x) := F(x
i
) +
F
x
(x
i
)(x x
i
) = 0 (3.34)
and we denote by x
i+1
the solution to (3.34) (assuming
F
x
(x
i
) is invertible).
Note that, from (3.34), x
i+1
is given by
x
i+1
= x
i

F
x
(x
i
)
1
F(x
i
) (3.35)
but it should not be computed that way but rather by solving the linear system (3.34)
(much cheaper than computing an inverse). It turns out that, under suitable conditions, x
i
converges very fast to a solution x

.
Exercise 3.35 Newtons method is invariant under non-singular linear ane transforma-
tion of the domain. Express this statement in a mathematically precise form, and prove
it.
Hence, in particular, if F : R
n
R
n
, Newtons method is invariant under scaling of the
individual components of x.
60 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
3.8 Newtons method
Theorem 3.11 Suppose F : V V is dierentiable, with locally Lipschitz-continuous
derivative. Let x

be such that F(x

) = 0 and suppose that


F
x
(x

) is invertible. Then there


exists > 0 such that, if |x
0
x

| < , x
i
x

. Moreover, convergence is Q-quadratic.


Proof. Let x
i
be such that
F
x
(x
i
) is invertible (by continuity, this holds close to x

; see note
below). Then, from (3.35),
|x
i+1
x

| = |x
i
x

F
x
(x
i
)
1
F(x
i
)| |
F
x
(x
i
)
1
| |F(x
i
) +
F
x
(x
i
)(x

x
i
)|,
where the induced norm is used for the (inverse) linear map. Let
1
,
2
, > 0 be such that
|
F
x
(x)
1
|
1
x B(x

, )
|F(x) +
F
x
(x)(x

x)|
2
|x x

|
2
x B(x

, )

2
< 1.
(Existence of
2
, for small enough follows from continuity of the second derivative of f and
from the fact stated right after Corollary B.2; can always be selected small enough that
the 3rd inequality holds.) It follows that
|x
i+1
x

|
1

2
|x
i
x

|
2
whenever x
i
B(x

, )

1

2
|x
i
x

| |x
i
x

|
so that, if x
0
B(x

, ), then x
i
B(x

, ) for all i, and thus


|x
i+1
x

|
1

2
|x
i
x

|
2
i (3.36)
and
|x
i+1
x

|
1

2
|x
i
x

| (
1

2
)
i
|x
0
x

| i .
Since
1

2
< 1, it follows that x
i
x

as i . In view of (3.36), convergence is


Q-quadratic.
Note. We have used the following fact: If L B(V, V ), V a Banach space, and L is invertible,
then every L

in a certain neighborhood of L is invertible, and the mapping L

L
1
is
continuous in that neighborhood.
We now turn back to our optimization problem
minf(x)[x V .
Since we are looking for points x

such that
f
x
(x

) = 0, we want to solve a system of


nonlinear equations and the theory just presented applies. The Newton iteration amounts
now to solving the linear system
f
x
(x
i
) +

2
f
x
2
(x
i
)(x x
i
) = 0
Copyright c _19932011, Andre L. Tits. All Rights Reserved 61
Unconstrained Optimization
i.e., nding a stationary point for the quadratic approximation to f
f(x
i
) +
f
x
(x
i
)(x x
i
) +
1
2
_

2
f
x
2
(x
i
)(x x
i
)
_
(x x
i
).
In particular, if f is quadratic, one Newton iteration yields the exact solution (which may
or may not be the minimizer). Quadratic convergence is achieved, e.g., if f is three times
continuously dierentiable and the 2nd order suciency condition holds at x

. We will
now show that we can obtain stronger convergence properties when the Newton iteration
is applied to a minimization problem. In particular, we want to achieve global convergence
(convergence for any initial guess x
0
). We can hope to achieve this because the optimization
problem has more structure than the general equation solving problem, as shown in the next
exercise.
Exercise 3.36 Exhibit a function F : R
n
R
n
which is not the gradient of any C
2
function f : R
n
R.
Remark 3.15 Global strong convexity of f (over all of R
n
) does not imply global conver-
gence of Newtons method to minimize f. (f = integral of the function plotted on Figure 3.5.
gives such an example).
Exercise 3.37 Exhibit a specic example, and test it, e.g., with Matlab.
x
1
x
0
x
2
F
divergent (although (x) > 0 x)
x
x
3
Figure 3.5:
Global convergence will be obtained by making use of a suitable step-size rule, e.g., the
Armijo rule.
Now suppose V = R
N
.
1
Note that the Newton direction h
N
(x) at some x is given by
h
N
(x) =
2
f(x)
1
f(x).
1
The same ideas apply for any Hlibert space, but some additional notation is needed.
62 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
3.8 Newtons method
When
2
f(x) 0, h
N
(x) is the gradient associated with the (local) inner product
u, v = u
T

2
f(x)v.
Hence, in such case, Newtons method is a special case of steepest descent.
Armijo - Newton
The idea is the following: replace the Newton iterate x
i+1
= x
i

2
f(x
i
)
1
f(x
i
) by a
suitable step in the Newton direction, i.e.,
x
i+1
= x
i

2
f(x
i
)
1
f(x
i
) (3.37)
with
i
suitably chosen. By controlling the length of each step, one hopes to prevent insta-
bility. Formula (3.37) ts into the framework of Algorithm 2, with
h(x) =
2
f(x)
1
f(x). (3.38)
Following Algorithm 2, we dene
i
in (3.37) via the Armijo step-size rule.
Note that if
2
f(x) is positive denite, h(x) is a descent direction. Global convergence with
an Armijo step, however, requires more (see (3.14)-(3.13)).
Exercise 3.38 Suppose f C
2
and suppose
2
f(x) 0 for all x x : f(x) f(x
0
).
Then for every x x : f(x) f(x
0
) there exists C > 0 and > 0 such that, for all x
close enuogh to x,
[[h
N
(x)[[ C[[f(x)[[
h
N
(x)
T
f(x) [[h
N
(x)[[ [[f(x)[[
so that the assumptions of Theorem 3.4 hold. Thus, any accumulation point of the sequence
x
k
generated by the Armijo-Newton algorithm is stationary. Moreover, this sequence does
have an accumulation point if and only if f has a global minimizer and, in such case, x
k
x

,
the unique global minimizer.
Armijo-Newton yields global convergence. However, the Q-quadratic rate may be lost since
nothing insures that the step-size
i
will be equal to 1, even very close to a solution x

.
However, as shown in the next theorem, this will be the case if the parameter is less
than 1/2.
Theorem 3.12 Consider Algorithm 2 with < 1/2 and h
i
= h(x
i
) with h(x) given by (3.38)
and suppose that x is an accumulation point of the sequence, with
2
f( x) > 0 (second order
suciency condition). Then there exists i
0
such that
i
= 1 for all i i
0
.
Exercise 3.39 Prove Theorem 3.12.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 63
Unconstrained Optimization
We now have a globally convergent algorithm, with (locally) a Q-quadratic rate (if f is thrice
continuously dierentiable). However, we needed a strong convexity assumption on f (on
any bounded set).
Suppose now that f is not strongly convex (perhaps not even convex). We noticed earlier
that, around a local minimizer, the strong convexity assumption is likely to hold. Hence,
we need to steer the iterate x
i
towards the neighborhood of such a local solution and then
use Armijo-Newton for local convergence. Such a scheme is called a stabilization scheme.
Armijo-Newton stabilized by a gradient method would look like the following.
Algorithm
Parameters. (0, 1/2), (0, 1)
Data. x
0
R
n
i = 0
while f(x
i
) ,= 0 do
if some suitable test is satised then
h
i
=
2
f(x
i
)
1
f(x
i
) (3.39)
else
h
i
= f(x
i
)
compute Armijo step size
i
x
i+1
= x
i
+
i
h
i

stop.
The above suitable test should be able to determine if x
i
is close enough to x

for the
Armijo-Newton iteration to work. (The goal is to force convergence while ensuring that
(3.39) is used locally.) Hence, it should at least include a test of positive deniteness of

2
f
x
2
(x
i
). Danger: oscillations!
3.9 Variable metric methods
Two major drawbacks of Newtons method are as follows:
1. for h
i
=
2
f(x
i
)
1
f(x
i
) to be a descent direction for f at x
i
, the Hessian
2
f(x
i
)
must be positive denite. (For this reason, we had to stabilize it with a gradient
method.)
2. second derivatives must be computed (
n(n+1)
2
of them!) and a linear system of equations
has to be solved.
The variable metric methods avoid these 2 drawbacks. The price paid is that the rate of
convergence is not quadratic anymore, but merely superlinear (as in the conjugate gradient
64 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
3.9 Variable metric methods
method). The idea is to construct increasingly better estimates S
i
of the inverse of the
Hessian, making sure that those estimates remain positive denite. Since the Hessian
2
f
is generally positive denite around a local solution, the latter requirement presents no
contradiction.
Algorithm
Data. x
0
R
n
, S
0
R
nn
, positive denite (e.g., S
0
= I)
while f(x
i
) ,= 0 do
h
i
= S
i
f(x
i
)
pick
i
arg min

f(x
i
+ h
i
)[ 0
x
i+1
= x
i
+
i
h
i
compute S
i+1
, positive denite, using some update formula

stop
If the S
i
are bounded and uniformly positive denite, all accumulation points of the sequence
constructed by this algorithm are stationary (why?).
Exercise 3.40 Consider the above algorithm with the step-size rule
i
= 1 for all i instead
of an exact search and assume f is strongly convex. Show that, if |S
i

2
f(x
i
)
1
| 0,
convergence is Q-superlinear (locally). [Follow the argument used for Newtons method. In
fact, if |S
i

2
f(x
i
)
1
| 0 fast enough, convergence may be quadratic.]
A number of possible update formulas have been suggested. The most popular one, due
independently to Broyden, Fletcher, Goldfarb and Shanno (BFGS) is given by
S
i+1
= S
i
+

i

T
i

T
i

i

S
i

T
i
S
i

T
i
S
i

i
where

= x
i+1
x
i

= f(x
i+1
) f(x
i
)
Convergence
If f is three times continuously dierentiable and strongly convex, the sequence x
i
gener-
ated by the BFGS algorithm converges superlinearly to the solution x

.
Remark 3.16
1. BFGS has been observed to perform remarkably well on non convex cost functions
2. Variable metric methods, much like conjugate gradient methods, use past information
in order to improve the rate of convergence. Variable metric methods require more
storage than conjugate gradient methods (an nn matrix) but generally exhibit much
better convergence properties.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 65
Unconstrained Optimization
Exercise 3.41 Justify the name variable metric as follows. Given a symmetric positive
denite matrix M, dene
[[x[[
M
= (x
T
Mx)
1/2
(= new metric)
and show that
inf
h
f(x + h) : [[h[[
S
1
i
= 1 (P

)
is achieved for h close to

h :=
S
i
f(x)
[[S
i
f(x)[[
S
1
i
for small. Specically, show that, given any

h ,=

h, with |

h|
S
1
i
= 1, there exists > 0 such that
f(x +

h) < f(x +

h) (0, ].
In particular, if

2
f
x
(x) > 0, the Newton direction is the direction of steepest descent in the
corresponding norm.
66 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
Chapter 4
Constrained Optimization
Consider the problem
minf(x) : x (P)
where f : V R is continuously Frechet dierentiable and is a subset of V , a normed
vector space. Although the case of interest here is when is not open, no such assumption
is made.
4.1 Abstract Constraint Set
In the unconstrained case, we obtained a rst order condition of optimality (
f
x
( x) = )
by approximating f in the neighborhood of x with its rst order expansion at x. Now that
the problem statement also includes a constraint set , we need some kind of rst order
approximation of valid near x. Perhaps the most obvious such approximation is the
radial cone.
Denition. The radial cone RC(x, ) to at x is dened by
RC(x, ) = h V :

t > 0 s.t. x + th t (0,

t] .
The following result is readily proved (as an extension of the proof of Theorem 3.1 of uncon-
strained minimization, or with a simplication of the proof of the next theorem).
Proposition 4.1 Suppose x

is a local minimizer for (P). Then


f
x
(x

)h 0 h cl(coRC(x

, ))
While this result is useful, it has a major drawback: When equality constraints are
present, RC(x

, ) is usually empty, so that theorem is vacuous. This motivates the intro-


duction of the tangent cone. But rst of all, let us dene what is meant by cone.
Denition 4.1 A set C V is a cone if x C implies x C for all > 0.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 67
Constrained Optimization
In other words given x C, the entire ray from the origin through x belongs to C (but
possibly not the origin itself). A cone may be convex or not.
Example 4.1 (Figure 4.1)
Exercise 4.1 Show that a cone C is convex if and only if x + y C for all , 0,
+ > 0, x, y C.
Exercise 4.2 Show that a cone C is convex if and only if
1
2
(x + y) C for all x, y C.
Show by exhibiting a counterexample that this statement is incorrect if cone is replaced
with set.
Exercise 4.3 Prove that radial cones are cones.
To address the diculty just encountered with radial cones, let us focus for a moment
on the case = (x, y) : y = f(x) (e.g., with x and y scalars), and with f nonlinear.
As just observed, the radial cone is empty in this situation. From freshman calculus we
do know how to approximate such set around x though: just replace f with its rst-order
Taylor expansion, yielding (x, y) : y = f( x) +
f
x
( x)(x x), where we have replaced the
curve with its tangent at x. Hence we have replace the radial-cone specication that a
ray belongs to the cone if short displacements along that ray yield points within , with
a requirement that short displacements along a candidate tangent direction to our curve
yield points little-o-close to the curve. Merging the two ideas yields the tangent cone, a
superset of the radial cone.
Denition 4.2 The tangent cone TC(x, ) to at x is dened by
TC(x, ) = h V : o(),

t > 0 s.t. x+th+o(t) t (0,

t],
o(t)
t
0 as t 0, t > 0 .
Note: some authors require that o() be continuous. This may yield a smaller set.
non convex
cone
0

2
0
convex
cone

a0
not a
cone

non convex cone

2
0

1

2

non convex
cone
0

3
Figure 4.1:
68 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
4.1 Abstract Constraint Set
Exercise 4.4 Show that tangent cones are cones.
Exercise 4.5 Show that
TC(x, ) = h V : o() s.t. x + th + o(t) t > 0,
o(t)
t
0 as t 0, t > 0 .
Exercise 4.6 Let : R be continuously dierentiable. Then, for all t,
d
dt
(t)
TC((t), ).
TC need not be convex, even if is dened by smooth equality and inequality constraints
(although A
_
g
x
(x)
_
and S(x), introduced below, are convex). Example: = x R
2
:
x
1
x
2
0. x

+ TC(x

, ) is an approximation to around x

.
Theorem 4.1 Suppose x

is a local minimizer for (P). Then


f
x
(x

)h 0 h cl(coTC(x

, )).
Proof. Let h TC(x

, ). Then
o() x

+ th + o(t) t 0 and lim


t0
o(t)
t
= 0. (4.1)
By denition of a local minimizer,

t > 0
f(x

+ th + o(t)) f(x

) t [0,

t] (4.2)
But, using the denition of derivative
f(x

+ th + o(t)) f(x

)
=
f
x
(x

)(th + o(t)) + o(th + o(t)) with


o(th)
t
0 as t 0
= t
f
x
(x

)h + o(t)
with o(t) =
f
x
(x

)o(t) + o(th + o(t)).


Hence
f(x

+ th + o(t)) f(x

) = t
_
f
x
(x

)h +
o(t)
t
_
0 t (0,

t]
so that
f
x
(x

)h +
o(t)
t
0 t (0,

t]
It is readily veried that
o(t)
t
0 as t 0. Thus, letting t 0, one obtains
f
x
(x

)h 0.
The remainder of the proof is left as an exercise.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 69
Constrained Optimization

C
x
*
+
T
C
(
x

,

)
*
x*
Figure 4.2:
Example 4.2 (Figure 4.2). In R
n
, if x

= solve (P), then f(x

)
T
h 0 for all h in the
tangent cone, i.e., f(x

) makes an angle of at least /2 with every vector in the tangent


cone, i.e., f(x

) C. (C is known as the normal cone.)


The necessary condition of optimality we just obtained is not easy to use. By considering
specic types of constraint sets , we hope to obtain a simple expression for TC, thus a
convenient condition of optimality.
We rst consider the equality constrained problem.
4.2 Equality Constraints - First Order Conditions
For simplicity, we now restrict ourselves to V = R
n
. Consider the problem
minf(x) : g(x) = 0 (4.3)
i.e.
minf(x) : x x : g(x) = 0
. .

where f : R
n
R and g : R
n
R
m
are both continuously Frechet dierentiable. Let
x

be such that g(x

) = 0. Let h TC(x

, ), i.e., suppose there exists a o() function such


that
g(x

+ th + o(t)) = 0 t
Since g(x

) = 0, we readily conclude that


g
x
(x

)h = 0. We investigate conditions under


which the converse holds, i.e., under which
TC(x

, ) =
_
h R
n
s.t.
g
x
(x

)h = 0
__
= A
_
g
x
(x

)
__
Remark 4.1 Unlike TC(x

, ), A
_
g
x
(x

)
_
depends not only on , but on the way is
formulated. For example
(x, y) s.t. x = 0 (x, y) s.t x
2
= 0
but A
_
g
x
(0, 0)
_
)) is not the same in both cases.
70 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
4.2 Equality Constraints - First Order Conditions
Example 4.3 (Figure 4.3)
1. m = 1 n = 2. Claim: g(x

)TC(x

, ) (why ?). Thus, from picture, TC(x

, ) =
A
_
g
x
(x

)
_
(assuming g(x

) ,= 0).
2. m = 2 n = 3. Again, TC(x

, ) = A
_
g
x
(x

)
_
, (again assuming g
1
(x

) ,= 0 ,=
g
2
(x

); convince yourself of this).


3. m = 2 n = 3. Here TC is a line but A
_
g
x
(x

)
_
is a plane (assuming g
1
(x

) ,= 0 ,=
g
2
(x

)). Thus TC(x

, ) ,= A
_
g
x
(x

)
_
. Note that x

could be a local minimizer


with f(x

) as depicted, although f(x

)
T
h < 0 for some h A
_
g
x
(x

)
_
.
(1) m=1 n=2
(2) m=2 n=3
(3) m=2 n=3

={xg(x)=0}
x +TC(x ,) *
x*

g (x)=0
1
g (x)=0
2
x*

g =0
2
g =0
1
f(x ) *
x*
*
Figure 4.3:
Proposition 4.2 c(coTC(x

, )) A
_
g
x
(x

)
_
.
Proof. Let h TC(x

, ). Then o()
x

+ th + o(t) t 0 (4.4)
Copyright c _19932011, Andre L. Tits. All Rights Reserved 71
Constrained Optimization
i.e.
g(x

+ th + o(t)) = g(x

) = 0 t 0 (4.5)
i.e.
0 = g(x

+ th + o(t)) g(x

) = t
_
g
x
(x

)h +
o(t, h)
t
_
t 0 (4.6)
Letting t 0, we get
g
x
(x

)h = 0, i.e. h A
_
g
x
(x

)
_
. (4.7)
The result follows by noting that A
_
g
x
(x

)
_
is closed and convex.
We want to determine under what conditions the two sets are equal, i.e., under what condi-
tions
A
_
g
x
(x

)
_
TC(x

, ).
Let h A
_
g
x
(x

)
_
, i.e.,
g
x
(x

)h = 0. We want to nd o() : R R
n
s.t.
s(t)

= x

+ th + o(t) t 0
Geometric intuition (see Figure 4.4) suggests that we could try to nd o(t) orthogonal to
h. (This does not work for the third example in Figure 4.3.). Since h A(
g
x
(x

)), we try
with o(t) in the range of
g
x
(x

)
T
, i.e.,

x*

x +th *
x + th
+ o(t)
*
Figure 4.4:
s(t) = x

+ th +
g
x
(x

)
T
u(t)
for some u(t) R
m
. We want to nd u(t) such that s(t) t i.e., such that g(x

+th +
g
x
(x

)
T
u(t)) = 0 t, and to see under what condition u exists and is a little o function.
We will make use of the implicit function theorem.
Theorem 4.2 (Implicit Function Theorem (IFT); see, e.g., [21]. Let F : R
m

R
nm
R
m
, m < n and x
1
R
m
, x
2
R
nm
be such that
(a) F C
1
72 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
4.2 Equality Constraints - First Order Conditions
(b) F( x
1
, x
2
) = 0
(c)
F
x
1
( x
1
, x
2
) is nonsingular.
Then > 0 and a function : R
nm
R
m

(i) ( x
2
) = x
1
(ii) C
1
in B( x
2
, )
(iii) F((x
2
), x
2
) = 0 x
2
B( x
2
, )
(iv)

x
2
(x
2
) = [

x
1
F((x
2
), x
2
)]
1
x
2
F((x
2
), x
2
) x
2
B( x
2
, ).
Interpretation (Figure 4.5)

x
2
x *
x
1

1
F(x ,x )=0
2
Figure 4.5:
Let n = 2, m = 1 i.e. x
1
R, x
2
R. [The idea is to solve the system of equations for x
1
(locally around ( x
1
, x
2
)]. Around x

, (

x
1
F(x

1
, x

2
) ,= 0) and x
1
is a well dened continuous
function of x
2
: x
1
= (x
2
). Around x, (

x
1
F( x
1
, x
2
) = 0) and x
1
is not everywhere dened
(specically, it is not dened for x
2
< x
2
) ((c) is violated). Note that (iv) is obtained by
dierentiating (iii) using the chain rule. We are now ready to prove the following theorem.
Exercise 4.7 Let A be a full rank mn matrix with m n. Then AA
T
is nonsingular. If
m > n, AA
T
is singular.
Theorem 4.3 Suppose that g(x

) = 0 and
g
x
(x

) is surjective (i.e., x

is a regular point).
Then TC(x

, ) = A
_
g
x
(x

)
_
. (Thus TC(x

, ) is convex and closed, indeed, it is a closed


subspace.)
Proof. We want to nd (t) such that s(t) := x

+ th +
g
x
(x

)
T
(t) satises
g(s(t)) = 0 t i.e.,
g
_
x

+ th +
g
x
(x

)
T
(t)
_
= 0 t
Copyright c _19932011, Andre L. Tits. All Rights Reserved 73
Constrained Optimization
Consider the function g : RR
m
R
m
g : (t, ) g
_
x

+ th +
g
x
(x

)
T

_
We now use the IFT on g with

t = 0, = 0. We have
(i) g(0, 0) = 0
(ii)
g

(0, 0) =
g
x
(x

)
g
x
(x

)
T
Hence
g

(0, 0) is non-singular and IFT applies i.e. : R R


m
and

t > 0 (0) = 0 and
g(t, (t)) = 0 t [

t,

t]
i.e.
g
_
x

+ th +
g
x
T
(x

)(t)
_
= 0 t [

t,

t].
Now note that a dierentiable function that vanishes at 0 is ao function if and only if its
derivative vanishes at 0. To exploit this fact, note that
d
dt
(t) =
_

g(t, (t))
_
1

t
g(t, (t)) t [

t,

t]
But, from the denition of g and since h A
_
g
x
(x

)
_

t
g(0, 0) =
g
x
(x

)h = 0
i.e.
d
dt
(0) = 0
and
(t) = (0) +
d
dt
(0)t + o(t) = o(t)
so that
g(x

+ th + o

(t)) = 0 t
with
o

(t) =
g
T
x
(x

)o(t)
which implies that h TC(x

, ).
Remark 4.2 Note that, in order for
g
x
(x

) to be full row rank, it is necessary that m n,


a natural condition for problem (4.3).
74 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
4.2 Equality Constraints - First Order Conditions
Remark 4.3 It follows from Theorem 4.3 that, when x

is a regular point, TC(x

, ) is
convex and closed, indeed, is a closed subspace. This subspace is typically referred to as the
tangent plane to at x

.
Remark 4.4 Suppose now g is scalar-valued. In such case, given R, the set x : g(x) =
is often referred to as the -level set of g. Then x

is a regular point if and only if


g(x

) ,= 0. It follows that, regardless of whether x

is regular or not, g(x

)
T
h = 0 for
all h in A(
g
x
(x

)), implying that g(x

)
T
h = 0 for all h in the tangent cone at x

to the
(unique) level set of g that includes x

. The gradient g(x

) is said-to-be normal to the


level set (or level surface, or level curve) through x

.
Remark 4.5 Also of interest is that connection between the gradient at some x R of a
scalar, continuously dierentiable function f : R
n
R and the normal to its graph
G := (x, z) R
n+1
: z = f(x).
Thus let x R
n
and z = f( x). Then ( x, z) is regular for the function z f(x) and a vector
of the form (g, 1) is orthogonal to the tangent plane at ( x, z) to G if and only if g = f( x).
Thus, the gradient at some point x of f is the horizontal projection of the downward normal
at that point to the graph of f, when the normal is scaled so that its vertical projection has
unit magnitude.
Exercise 4.8 Prove the claims made in Remark 4.5.
We are now ready to state and prove the main optimality condition.
Theorem 4.4 Suppose x

is a local minimizer for (4.3) and suppose that


g
x
(x

) is full row
rank. Then
f
x
(x

)h = 0 h A
_
g
x
(x

)
_
Proof. From Theorem 4.1,
f
x
(x

)h 0 h TC(x

, ) (4.8)
so that, from Theorem 4.3
f
x
(x

)h 0 h A
_
g
x
(x

)
_
. (4.9)
Now obviously h A
_
g
x
(x

)
_
implies h A
_
g
x
(x

)
_
. Hence
f
x
(x

)h = 0 h A
_
g
x
(x

)
_
(4.10)
Copyright c _19932011, Andre L. Tits. All Rights Reserved 75
Constrained Optimization
Corollary 4.1 (Lagrange multipliers)
Under same assumptions, R
m

f(x

) +
g
x
(x

)
T
= 0 (4.11)
i.e.
f(x

) +
m

j=1

j
g
j
(x

) = 0 (4.12)
Proof. From the theorem above f(x

) A
_
g
x
(x

)
_

= A
_
g
x
(x

)
_

= 1
_
g
x
(x

)
T
_
,
i.e.,
f(x

) =
m

j=1

j
g
j
(x

) (4.13)
for some

1
, . . . ,

m
. The proof is complete if one sets
j
=

j
, j = 1, 2, . . . , m.
Remark 4.6 Regularity of x

is not necessary in order for (4.11) to hold. For example,


consider cases where two components of g are identical (in which case there are no regular
points).
Corollary 4.2 Suppose that x

is a local minimizer for (4.3) (without full rank assumption).


Then R
m+1
, = (
0
,
1
, . . . ,
m
) ,= 0, such that

0
f(x

) +
m

j=1

j
g
j
(x

) = 0 (4.14)
Proof. If
g
x
(x

) is not full rank, then (4.14) holds with


0
= 0 (g
j
(x

) are linearly
dependent). If
g
x
(x

) is full rank, (4.14) holds with


0
= 1, from Corollary 4.1.
Remark 4.7
1. How does the optimality condition (4.12) help us solve the problem? Just remember
that x

must satisfy the constraints, i.e.,


g(x

) = 0 (4.15)
Hence we have a system of n + m equations with n + m unknowns (x

and ). Now,
keep in mind that (4.12) is only a necessary condition. Hence, all we can say is that,
if there exists a local minimizer satisfying the full rank assumption, it must be among
the solutions of (4.12)+(4.15).
76 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
4.3 Equality Constraints Second Order Conditions
2. Solutions with
0
= 0 are degenerate in the sense that the cost function does not enter
the optimality condition.
Exercise 4.9 Find the point in R
2
that is closest to the origin and also satises
(x
1
a)
2
A
2
+
(x
2
b)
2
B
2
= 1 .
Lagrangian function
If one denes L : R
n
R
m
R by
L(x, ) = f(x) +
m

j=1

j
g
j
(x)
then, the optimality conditions (4.12)+(4.15) can be written as:

R
m
s.t.

x
L(x

) = 0 , (4.16)

L(x

) = 0 . (4.17)
L is called the Lagrangian for problem (4.3).
4.3 Equality Constraints Second Order Conditions
Assume V = R
n
. In view of the second order conditions for the unconstrained problems,
one might expect a second order necessary condition of the type
h
T

2
f(x

)h 0 h A
_
g
x
(x

)
_
(4.18)
since it should be enough to consider directions h in the tangent plane to at x

. The
following exercise show that this is not true in general.
Exercise 4.10 Consider the problem
minf(x, y) x
2
+ y : g(x, y) y kx
2
= 0 , k > 1 .
Show that (0, 0) is the unique global minimizer and that it does not satisfy (4.18). ((0,0)
does not minimize f over the tangent cone (=tangent plane).)
The correct second order conditions will involve the Lagrangian.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 77
Constrained Optimization
Theorem 4.5 (2nd order necessary condition). Suppose x

is a local minimizer for a 4.3)


and suppose that
g
x
(x

) is surjective. Also suppose that f and g are twice continuously


dierentiable. Then there exists R
m
such that (4.16) holds and
h
T

2
x
L(x

, )h 0 h A
_
g
x
(x

)
_
(4.19)
Proof. Let h A
_
g
x
(x

)
_
= TC(x

, ) (since
g
x
(x

) is full row rank). Then o() s.t. x

+
th + o(t) s(t) t 0 i.e.
g(s(t)) = 0 t 0 (4.20)
and, since x

is a local minimizer, for some



t > 0
f(s(t)) f(x

) t [0,

t] (4.21)
We can write t [0,

t]
0 f(s(t))f(x

) = f(x

)
T
(th+o(t))+
1
2
(th+o(t))
T
_

2
f(x

)(th + o(t))
_
+o
2
(t) (4.22)
and, for j = 1, 2, . . . , m,
0 = g
j
(s(t)) g
j
(x

) = g
j
(x

)
T
(th + o(t)) +
1
2
(th + o(t))
T
_

2
g
j
(x

)(th + o(t))
_
+ o
j
2
(t)
(4.23)
where
o
2
(t)
t
2
0, as t 0,
o
j
2
(t)
t
2
0 as t 0 for j = 1, 2, . . . , m. (Note that the rst term in
the RHS of (4.22) is generally not 0, because of the o term, and even dominates the second
term; this is why conjecture (4.18) is incorrect.) We have shown (1st order condition) that
there exists R
m
such that
f(x

) +
m

j=1

j
g
j
(x

) = 0
Hence, multiplying the jth equation in (4.23) by
j
and adding all of them, together with
(4.22), we get
0 L(s(t), )L(x

, ) = L(x

, )
T
(th+o(t))+
1
2
(th+o(t))
T

2
L(x

, )(th+o(t))+ o
2
(t)
yielding
0
1
2
(th + o(t))
T

2
L(x

, )(th + o(t)) + o
2
(t)
with
o
2
(t) = o
2
(t) +
m

j=1

j
o
j
2
(t)
which can be rewritten as
0
t
2
2
h
T

2
L(x

, )h + o
2
(t) with
o
2
(t)
t
2
0 as t 0.
78 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
4.3 Equality Constraints Second Order Conditions
Dividing by t
2
and letting t 0 (t > 0), we obtain the desired result.
Just like in the unconstrained case, the above proof cannot be used directly to obtain a
suciency condition, by changing to < and proceeding backwards. In fact, an additional
diculty here is that all we would get is
f(x

) < f(s(t)) for any h A


_
g
x
(x

)
_
and s(t) = x

+ th + o
h
(t),
for some little o function o
h
and for all t (o,

t
h
] for some

t
h
> 0. It is not clear whether
this would ensure that
f(x

) < f(x) x B(x

, ) x

, for some > 0.


It turns out that it does, due to the following lemma.
Lemma 4.1 Suppose that x
k
x

, with g(x
k
) = g(x

) = 0 and
x
k
x

[[x
k
x

[[
s

for some s

.
Then
g
x
(x

)s

= 0. (The set of such limits is often referred to as the sequential cone to


at x

. Thus the sequential cone is included in A(


g
x
(x

)).)
Proof. Let x
k
= x

+
k
s
k
, with
k
0 as k and [[s
k
[[ = 1 k. Since g(x

) = 0, we
have
0 = g(x

+
k
s
k
) =
k
g
x
(x

) s
k
+ o(
k
)
=
k

g
x
(x

) s
k
+
o(
k
)

Since s
k
s

, and since
o(
k
)

k
0 as k , this implies
g
x
(x

)s

= 0, i.e., s

A
_
g
x
(x

)
_
.
We now state and prove the 2nd order sucient condition of optimality.
Theorem 4.6 (2nd order suciency condition). Suppose that f and g are twice continu-
ously dierentiable. Suppose that x

R
n
is such that
(i) g(x

) = 0
(ii) (4.16) is satised for some

R
m
(iii) h
T

2
L(x

)h > 0 h A
_
g
x
(x

)
_
, h ,= 0
Then x

is a strict local minimizer for (4.3)


Copyright c _19932011, Andre L. Tits. All Rights Reserved 79
Constrained Optimization
Proof. By contradiction. Suppose x

is not a strict local minimizer. Then > 0, x


B(x

, ), x ,= x

s.t. g(x) = 0 and f(x) f(x

) i.e., a sequence s
k
R
n
, [[s
k
[[ = 1 and
a sequence of positive scalars
k
0
g(x

+
k
s
k
) = 0 (4.24)
f(x

+
k
s
k
) f(x

) (4.25)
We can write
0 f(x
k
) f(x

) =
k
f(x

)
T
s
k
+
1
2

2
k
s
T
k

2
f(x

)s
k
+ o
2
(
k
s
k
)
0 = g
j
(x
k
) g
j
(x

) =
k
g
j
(x

)
T
s
k
+
1
2

2
k
s
T
k

2
g
j
(x

)s
k
+ o
j
2
(
k
s
k
)
Multiplying by the multipliers

j
given by (ii) and adding, we get
0
1
2

2
k
s
T
k

2
L(x

, )s
k
+ o
2
(
k
s
k
)
i.e.
1
2
s
T
k

2
L(x

, )s
k
+
o
2
(
k
s
k
)

2
k
0 k (4.26)
Since [[s
k
[[ = 1 k, s
k
lies in a compact set and s

and K s.t. s
k
K
s

. Without loss
of generality, assume s
k
s

. Taking k in (4.26), we get


(s

)
T

2
L(x

, )s

0.
Since from the lemma, s

A
_
g
x
(x

)
_
, this is a contradiction.
4.4 Inequality Constraints First Order Conditions
Consider the problem
minf
0
(x) : f(x) 0 (4.27)
where f
0
: R
n
R and f : R
n
R
m
are continuously dierentiable. Recall that, in the
equality constrained case (see (4.3)), we obtained 2 rst order conditions: a strong condition
such that f(x

) +
g
x
(x

)
T
= 0 (4.28)
subject to the assumption that
g
x
(x

) is full rank, and a weaker condition


(
0
, ) ,= 0 such that
0
f(x

) +
g
x
(x

)
T
= 0. (4.29)
80 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
4.4 Inequality Constraints First Order Conditions
For the inequality constraint case, we shall rst obtain a weak condition (Fritz John condi-
tion) analogous to (4.29). Then we will investigate when a strong condition (Karush-Kuhn-
Tucker condition) holds.
We use the following notation. For x R
n
J(x) = j 1, . . . , m : f
j
(x) 0 (4.30)
J
0
(x) = 0 J(x) (4.31)
i.e., J(x) is the set of indices of active or violated constraints at x whereas J
0
(x) includes
the index of the cost function as well.
Exercise 4.11 Let s R
m
be a vector of slack variables. Consider the 2 problems
(P
1
) min
x
f
0
(x) : f
j
(x) 0, j = 1, . . . , m
(P
2
) min
x,s
f
0
(x) : f
j
(x) + (s
j
)
2
= 0, j = 1, . . . , m
(i) Prove that if ( x, s) solves (P
2
) then x solves (P
1
)
(ii) Prove that if x solves (P
1
) then ( x, s) solves (P
2
) where
s
j
=
_
f
j
( x)
(iii) Use Lagrange multipliers rule and part (ii) to prove (carefully) the following weak result.
If x solves (P
1
) and f
j
( x) : j J( x) is a linearly-independent set, then there exists a
vector R
m
such that
f
0
( x) +
f
x
( x)
T
= 0 (4.32)

j
= 0 if f
j
( x) < 0 j = 1, 2, . . . , m (4.33)
This result is weak because: (i) there is no constraint on the sign of , (ii) the results can be
shown to hold under a weaker regularity condition.
To obtain stronger conditions, we now proceed, as in the case of equality constraints, to
characterize TC(x

, ). For x

we rst dene the set of rst-order strictly feasible


directions

S
f
(x

) := h : f
j
(x

)
T
h < 0 j J(x

).
It is readily checked that S
f
(x

) is a convex cone. Further, as shown next, it is a subset of


the radial cone, hence of the tangent cone.
Theorem 4.7 If x

(i.e., f(x

) 0),
Copyright c _19932011, Andre L. Tits. All Rights Reserved 81
Constrained Optimization

S
f
(x

) RC(x

, ) .
Proof. Let h

S
f
(x

). For j J(x

), we have, since f
j
(x

) = 0,
f
j
(x

+ th) = f
j
(x

+ th) f
j
(x

) = t f
j
(x

)
T
h + o
j
(t) (4.34)
= t
_
f
j
(x

)
T
h +
o
j
(t)
t
_
. (4.35)
Since f
j
(x

)
T
h < 0, there exists

t
j
> 0 such that
f
j
(x

)
T
h +
o
j
(t)
t
< 0 t (0,

t
j
] (4.36)
and, with

t = min

t
j
: j J(x

) > 0,
f
j
(x

+ th) < 0 j J(x

), t (0,

t]. (4.37)
For j 1, . . . , mJ(x

), f
j
(x

) < 0, thus, by continuity



t > 0 s.t.
f
j
(x

+ th) < 0 j 1, . . . , mJ(x

), t (0,

t]
Thus
x

+ th t (0, min(

t,

t)]
Theorem 4.8 Suppose x

is a local minimizer for (P). Then


f
0
(x

)
T
h 0 for all h s.t. f
j
(x

)
T
h < 0 j J(x

)
or, equivalently
,h s.t. f
j
(x

)
T
h < 0 j J
0
(x

).
Proof. Follows directly from Theorems 4.7 and 4.1.
Denition 4.3 A set of vectors a
1
, . . . , a
k
R
n
is said to be positively linearly independent
if

k
i=1

i
a
i
= 0

i
0, i = 1, . . . , k
_
implies
i
= 0 i = 1, 2, . . . , k
If the condition does not hold, they are positively linearly dependent.
Note that, if a
1
, . . . , a
k
are linearly independent, then they are positively linearly indepen-
dent.
The following proposition gives a geometric interpretation to the necessary condition just
obtained.
82 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
4.4 Inequality Constraints First Order Conditions
Theorem 4.9 Given a
1
, . . . , a
k
, a nite collection of vectors in R
n
, the following three
statements are equivalent
(i) ,h R
n
such that a
T
j
h < 0 j = 1, . . . , k
(ii) 0 coa
1
, . . . , a
k

(iii) the a
j
s are positively linearly dependent.
Proof
(i)(ii): By contradiction. If 0 , coa
1
, . . . , a
k
then by the separation theorem (see Ap-
pendix B) there exists h such that
v
T
h < 0 v coa
1
, . . . , a
k
.
In particular a
T
j
h < 0 for j = 1, . . . , k. This contradicts (i)
(ii)(iii): If 0 coa
1
, . . . , a
k
then (see Exercise B.24 in Appendix B), for some
j
,
0 =
k

j=1

j
a
j

j
0
k

j
= 1
Since positive numbers which sum to 1 cannot be all zero, this proves (iii).
(iii)(i): By contradiction. Suppose (iii) holds, i.e.,

i
a
i
= 0 for some
i
0, not all
zero, and there exists h such that a
T
j
h < 0, j = 1, . . . , k. Then
0 =
_
k

j
a
j
_
T
h =
k

j
a
T
j
h.
But since the
j
s are non negative, not all zero, this is a contradiction.
Corollary 4.3 Suppose x

is a local minimizer for (4.27). Then


0 cof
j
(x

) : j J
0
(x

)
Corollary 4.4 (F. John conditions). Suppose x

is a local minimizer for (4.27). Then there


exist
0
,

1
, . . . ,
m
, not all zero such that
(i)
0
f
0
(x

) +
m

j=1

j
f
j
(x

) = 0
(ii) f
j
(x

) 0 j = 1, . . . , m
(iii)
j
0 j = 0, 1, . . . , m
(iv)
j
f
j
(x

) = 0 j = 1, . . . , m
Copyright c _19932011, Andre L. Tits. All Rights Reserved 83
Constrained Optimization
Proof. From (iii) in previous theorem,
j
, j J
0
(x

),
j
0, not all zero such that

jJ
0
(x

j
f
j
(x

) = 0.
By dening
j
= 0 for j , J
0
(x

) we obtain (i). Finally, (ii) just states that x

is feasible,
(iii) directly follows and (iv) follows from
j
= 0 j , J
0
(x

).
Remark 4.8 Condition (iv) in Corollary 4.4 is called complementary slackness. In view of
conditions (ii) and (iii) it can be equivalently stated as
(

)
T
f(x

) =
m

j
f
j
(x

) = 0
The similarity between the above F. John conditions and the weak conditions obtained for
the equality constrained case is obvious. Again, if
0
= 0, the cost f
0
does not enter the
conditions at all. These conditions are degenerate.
We are now going to try and obtain conditions for the optimality conditions to be non-
degenerate, i.e. for the rst multiple
0
to be nonzero (in that case, it can be set to 1
by dividing through by
0
). The resulting optimality conditions are called Karush-Kuhn-
Tucker (or Kuhn-Tucker) optimality conditions. The general condition for
0
,= 0 is called
Kuhn-Tucker constraint qualication (KTCQ). But before considering it in some detail, we
give a simpler (but stronger) condition.
If the gradients of the active constraints are linearly independent, a strong result (
0
,= 0)
holds.
Proposition 4.3 Suppose x

is a local minimizer for (4.27) and suppose f


j
(x

) : j
J(x

) is a positively linearly independent set of vectors. Then the Fritz John conditions
hold with
0
,= 0.
Proof. By contradiction. Suppose
0
= 0. Then, from Corollary 4.4 above,
j
0, j =
1, . . . , m, not all zero such that
m

j=1

j
f
j
(x

) = 0
and, since
j
= 0 j , J
0
(x

),

jJ(x

j
f
j
(x

) = 0
which contradicts the positive linear independence assumption.
Note: linear independence implies uniqueness of the multipliers.
84 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
4.4 Inequality Constraints First Order Conditions
x *

Figure 4.6:
We now investigate weaker conditions under which a strong result holds. It turns out to
be the case whenever S
f
(x

), dened by
S
f
(x

) := h : f
j
(x

)
T
h 0 j J(x

) (4.38)
is exactly the convex hull of the closure of the tangent cone. (It is readily veried that S
f
(x

)
is a closed convex cone.)
Remark 4.9 The inclusion

S
f
(x

) TC(x

, ) (Theorem 4.7) does not imply that


S
f
(x

) cl(coTC(x

, )) (4.39)
since, in general
cl(

S
f
(x

)) S
f
(x

) (4.40)
but equality in (4.40) need not hold. (As noted below, it holds if and only if

S
f
(x

) is
nonempty.) Condition (4.39) is known as the KTCQ condition (see below).
The next example gives an instance of Kuhn-Tucker constraint qualication, to be intro-
duced later.
Example 4.4 In the rst three examples below, the gradients of the active constraints are
not positively linearly independent. In some cases though, KTCQ does hold.
(1) (Figure 4.6)
f
1
(x) x
2
x
2
1
f
2
(x) x
2
x

= (0, 0)
_

S
f
(x

) =
S
f
(x

) = h s.t. h
2
= 0 = TC(x

, )
but (4.39) holds anyway
(2) (Figure 4.7)
f
1
(x) x
2
x
3
1
f
2
(x) = x
2
x

= (0, 0)
_

S
f
(x

) =
S
f
(x

) = h s.t. h
2
= 0
TC(x

, ) = h : h
2
= 0, h
1
0
and (4.39) does not hold
If f
2
is changed to x
4
1
x
2
, KTCQ still does not hold, but with the cost function f
0
(x) = x
2
,
the KKT conditions do hold! (Hence KTCQ is sucent but not necessary in order for KKT
to hold. Also see Exercise 4.17 below.)
Copyright c _19932011, Andre L. Tits. All Rights Reserved 85
Constrained Optimization

Figure 4.7:
f (x)=0 0 above
1
f (x)=0 0 below
2
x*
Figure 4.8:
(3) Example similar to that given in the equality constraint case; TC(x

, ) and S
f
(x

) are
the same as in the equality case (Figure 4.8). KTCQ does not hold.
(4) (x, y, z) : z(x
2
+2y
2
) 0, z(2x
2
+y
2
) 0. At (0, 0, 0), the gradients are positively
linearly independest, though they are not linearly independent.

S
f
(x

) ,= , so that KTCQ
does hold.
The next example shows that the reverse inclusion always holds. That inclusion is anal-
ogous to the inclusion cl(coTC(x

, )) A
_
g
x
(x

)
_
in the equality-constrained case.
Exercise 4.12 If x

, then
cl(coTC(x

, )) S
f
(x

).
Denition 4.4 The Kuhn-Tucker constraint qualication (KTCQ) is satised at x if
cl(coTC( x, )) = h : f
j
( x)
T
h 0 j J( x) (= S
f
( x)) (4.41)
Recall (Exercise 4.12) that the left-hand side of (4.41) is always a subset of the right-hand
side. Thus KTCQ is satised if RHSLHS. In particular this holds whenever

S
f
(x) ,= (see
Proposition 4.7 below).
Remark 4.10 The inequality constraint g(x) = 0 can be written instead as the pair of
inequalities g(x) 0 and g(x) 0. If there are no other constraints, the we get S
f
( x) =
A
_
g
x
(x

)
_
, and it is readily checked that, if x

is regular, then KTCQ does hold. However,


none of the sucient conditions we have discussed (for KTCQ to hold) are satised in such
case!
Exercise 4.13 (due to H.W. Kuhn). Obtain the sets S
f
(x

),

S
f
(x

) and TC(x

, ) at min-
imizer x

for the following examples (both have the same ): (i) minimize (x
1
) subject
to x
1
+ x
2
1 0, x
1
+ 2x
2
1 0, x
1
0, x
2
0, (ii) minimize (x
1
) subject to
(x
1
+x
2
1)(x
1
+2x
2
1) 0; x
1
0, x
2
0. In each case, indicate whether KTCQ holds.
86 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
4.4 Inequality Constraints First Order Conditions
Exercise 4.14 In the following examples, exhibit the tangent cone at (0, 0) and determine
whether KTCQ holds
(i) = (x, y) : (x 1)
2
+ y
2
1 0, x
2
+ (y 1)
2
1 0
(ii) = (x, y) : x
2
+ y 0, x
2
y 0.
(iii) = (x, y) : x
2
+ y 0, x
2
y 0, x 0
(iv) = (x, y) : xy 0.
The following necessary condition of optimality follows trivially.
Proposition 4.4 Suppose x

is a local minimizer for (P) and KTCQ is satised at x

. Then
f
0
(x

)
T
h 0 h S
f
(x

) i.e. h s.t. f
j
(x

)
T
h 0 j J(x

) (4.42)
Proof. Follows directly from (4.41) and the rst theorem in this chapter.
Remark 4.11 The above result appears to be only very slightly stronger than the theorem
right after Exercise 4.12. It turns out that this is enough to ensure
0
,= 0. This result
follows directly from Farkass Lemma.
Proposition 4.5 (Farkass Lemma). Consider a
1
, . . . , a
k
, b R
n
. Then b
T
x 0 x
x : a
T
i
x 0, i = 1, . . . , k if and only if

i
0, i = 1, 2, . . . , k s.t. b =
k

i=1

i
a
i
Proof. (): Obvious. (): Consider the set
C = y : y =
k

i=1

i
a
i
,
i
0, i = 1, . . . , k .
Exercise 4.15 C is a closed convex cone.
We now proceed by contra-position. Suppose b , C. Since b is convex and compact and
C is convex and closed, they can be strictly separated, i.e., x, s.t.
x
T
b > > x
T
v v C. (4.43)
Taking v = 0 i yields
> 0
Copyright c _19932011, Andre L. Tits. All Rights Reserved 87
Constrained Optimization
so that
x
T
b > 0.
To complete the proof, we now establish that
x
T
v 0 v C.
(Indeed, this implies that a
T
i
x 0 i, in contradiction with the premise.) Indeed, suppose
it is not the case, i.e., suppose v

C s.t.
x
T
v

= > 0
Setting v =

( C) then yields
x
T
v =
This contradicts (4.43).
Theorem 4.10 (Karush-Kuhn-Tucker). Suppose x

is a local minimizer for (4.28) and


KTCQ holds at x

. Then there exists

R
m
such that
f
0
(x

) +
m

i=1

j
f
j
(x

) = 0

j
0 j = 1, . . . , m
f
j
(x) 0 j = 1, . . . , m

j
f
j
(x

) = 0 j = 1, . . . , m
Proof. From (4.42) and Farkass Lemma
j
, j J(x

) such that
f
0
(x

) +

jJ(x

j
f
j
(x

) = 0
with
j
0, j J(x

)
Setting
j
= 0 for j , J(x

) yields the desired results.


Remark 4.12 An interpretation of Farkass Lemma is that the closed convex cone C and
the closed convex cone
D = x : a
T
i
x 0, i = 1, . . . , k
are dual of each other, in the sense that
D = x : y
T
x 0, y C,
and
C = y : x
T
y 0, x D.
88 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
4.4 Inequality Constraints First Order Conditions
This is often written as
D = C

, C = D

.
In the case of a subspace S (subspaces are cones), the dual cone is simply the orthogonal
complement, i.e., S

= S

(check it). Hence the fundamental property of linear maps L


A(L) = 1(L

can be expressed in the notation of cone duality as


A(L) = 1(L

and our proofs of Corollary 4.1 (equality constraint case) and Theorem 4.10 (inequality
constraint case) from Theorem 4.4 and Proposition 4.4 respectively, are analogous.
We pointed out earlier a sucient condition for
0
,= 0 in the Fritz John conditions. We
now consider two other important sucient conditions and we show that they imply KTCQ.
Denition 4.5 h : R
n
R is ane if h(x) = a
T
x + b.
Proposition 4.6 Suppose the f
j
s are ane. Then = x : f
j
(x) 0, j = 1, 2, . . . , m
satises KTCQ at any x .
Exercise 4.16 Prove Proposition 4.6.
The next exercise show that KTCQ (a property of the description of the constraints) is not
necessary in order for KKT to hold for some objective function.
Exercise 4.17 Consider again the optimization problems in Exercise 4.13. In both cases
(i) and (ii) check whether the Kuhn-Tucker optimality conditions hold. Then repeat with the
cost function x
2
instead of x
1
. (This substitution clearly does not aect whether KTCQ
holds!)
Remark 4.13 Note that, as a consequence of this, the strong optimality condition holds for
any equality constrained problem whose constraints are all ane (with no regular point
assumption).
We know that it always holds
cl(TC(x

, )) S
f
(x

) and cl(

S
f
(x

)) cl(TC(x

, )) .
Thus, a sucient condition for KTCQ is
S
f
(x

) cl(

S
f
(x

)).
The following proposition gives (clearly, the only instance except for the trivial case when
both S
f
(x

) and

S
f
(x

) are empty) when this holds.


Copyright c _19932011, Andre L. Tits. All Rights Reserved 89
Constrained Optimization
Proposition 4.7 (Mangasarian-Fromovitz) Suppose that there exists

h R
n
such that
f
j
( x)
T

h < 0 j J( x), i.e., suppose



S
f
( x) ,= . Then S
f
( x) cl(

S
f
( x)) (and
thus, KTCQ holds at x).
Proof. Let h S
f
( x) and let h
i
=
1
i

h + h i = 1, 2, . . . Then
f
j
( x)
T
h
i
=
1
i
f
j
( x)
T

h
. .
<0
+f
j
( x)
T
h
. .
0
< 0 j J( x) (4.44)
so that h
i


S
f
( x) i. Since h
i
h as i , h cl(

S
f
( x)).
Exercise 4.18 Suppose the gradients of active constraints are positively linearly indepen-
dent. Then KTCQ holds. (Hint: h s.t. f
j
(x)
T
h < 0 j J(x).)
Exercise 4.19 Suppose the gradients of the active constraints are linearly independent.
Then the KKT multipliers are unique.
Exercise 4.20 (First order sucient condition of optimality.) Suppose that x

is feasible
and that, among all constraints, there exist n (where n is the dimension of the space) con-
straints that (i) are active at x

, (ii) have linear independent gradients at x

, and (iii) have


strictly positive associated multipliers in the KKT conditions of optimality. Then x

is a
strict local minimizer.
4.5 Mixed Constraints First Order Conditions
We consider the problem
minf
0
(x) : f(x) 0, g(x) = 0 (4.45)
with
f
0
: R
n
R, f : R
n
R
m
, g : R
n
R

, all C
1
We rst obtain an extended Karush-Kuhn-Tucker condition. We dene
S
f,g
(x

) = h : f
j
(x

)
T
h 0 j J(x

),
g
x
(x

)h = 0
As earlier, KTCQ is said to hold at x

if
S
f,g
(x

) = cl(coTC(x

, ))
Exercise 4.21 Again S
f,g
(x

) cl(coTC(x

, )) always holds.
Fact. If g
j
(x), j = 1, . . . , f
j
(x), j J(x) is a linearly independent set of vectors,
then KTCQ holds at x.
90 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
4.5 Mixed Constraints First Order Conditions
Theorem 4.11 (extended KKT conditions). If x

is a local minimizer for (4.45) and KTCQ


holds at x

then

R
m
,

0
f(x

) 0, g(x

) = 0
f
0
(x

) +
m

j=1

j
f
j
(x

) +

j=1

j
g
j
(x

) = 0

j
f
j
(x

) = 0 j = 1, . . . , m
Proof (sketch)
Express S
f,g
(x

) by means of pairs of inequalities of the form g


j
(x

)
T
h 0, g
j
(x

)
T
h
0, and use Farkass lemma.
Without the KTCQ assumption, the following (weak) result holds (see, e.g., [5], sec-
tion 3.3.5).
Theorem 4.12 (extended F. John conditions). If x

is a local minimizer for (4.45) then

j
, j = 0, 1, 2, . . . , m and
j
j = 1, . . . , , not all zero, such that

j
0 j = 0, 1, . . . , m
f(x

) 0, g(x

) = 0
m

j=0

j
f
j
(x

) +

j=1

j
g
j
(x

) = 0

j
f
j
(x

) = 0 j = 1, . . . , m (complementary slackness)
Note the constraint on the sign of the
j
s (but not of the
j
s) and the complementary
slackness condition for the inequality constraints.
Exercise 4.22 The argument that consists in splitting again the equalities into sets of 2
inequalities and expressing the corresponding F. John conditions is inappropriate. Why?
Theorem 4.13 (Convex problems). Consider problem (4.45). Suppose that f
j
, j =
0, 1, 2, . . . , m are convex and that g
j
, j = 1, 2, . . . , are ane. Under those conditions,
if x

is such that

R
m
,

which, together with x

, satisfy the KKT conditions,


then x

is a global minimizer for (4.45).


Proof. Dene : R
n
R as
(x) = f
0
(x) +
m

j=1

j
f
j
(x) +

j=1

j
g
j
(x)
with

and

as given in the theorem.


Copyright c _19932011, Andre L. Tits. All Rights Reserved 91
Constrained Optimization
(i) () is convex (prove it)
(ii) (x

) = f
0
(x

) +
m

j=1

j
f
j
(x

) +

j=1

j
g
j
(x

) = 0
since (x

) is a KKT triple.
(i) and (ii) imply that x

is a global minimizer for , i.e.,


(x

) (x) x R
n
in particular,
(x

) (x) x
i.e.
f
0
(x

) +
m

j=1

j
f
j
(x

) +

j=1

j
g
j
(x

) f
0
(x) +
m

j=1

j
f
j
(x) +

j=1

j
g
j
(x) x .
Since (x

) is a KKT triple this simplies to


f
0
(x

) f
0
(x) +
m

j=1

j
f
j
(x) +

j=1

j
g
j
(x) x
and, since for all x , g(x) = 0, f(x) 0 and

0,
f
0
(x

) f
0
(x) x .
Exercise 4.23 Under the assumptions of the previous theorem, if f
0
is strictly convex, then
x

is the unique global minimizer for (P).


Remark 4.14 Our assumptions require that g be ane (not just convex). In fact, what we
really need is that (x) be convex and this may not hold if g is merely convex. For example
(x, y) : x
2
y = 0 is obviously not convex.
4.6 Mixed Constraints Second order Conditions
Theorem 4.14 (necessary condition). Suppose that x

is a local minimizer for (4.45) and


suppose that f
j
(x

), j J(x

) g
j
(x

), j = 1, . . . , is a linearly independent set of


vectors. Then there exist

R
n
and

such that the KKT conditions hold and


h
T

2
L(x

)h 0 h h :
g
x
(x

)h = 0, f
j
(x

)
T
h = 0 j J(x

) (4.46)
with L(x, , )

= f
0
(x) +
m

j=1

j
f
j
(x) +

j=1

j
g
j
(x).
92 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
4.6 Mixed Constraints Second order Conditions
Proof. It is clear that x

is also a local minimizer for the problem


minimize f
0
(x) s.t. g(x) = 0
f
j
(x) = 0 j J(x

) .
The claims is then a direct consequence of the second order necessary condition of optimality
for equality constrained problems.
Remark 4.15
1. The linear independence assumption is stronger than KTCQ. It insures uniqueness of
the KKT multipliers (prove it).
2. Intuition may lead to believe that (4.46) should hold
h S
f,g
(x

) := h :
g
x
(x

)h = 0, f
j
(x

)
T
h 0 j J(x

) .
The following example shows that this is not true.
minlog(1 + x) : x 0 (with x R)
(Note that a rst order suciency condition holds for this problem: see Exercises 4.20
and 4.24)
There are a number of (non-equivalent) secobd-order sucient conditions (SOSCs) for prob-
lems with mixed (or just inequality) constraints. The one stated below strikes a good tradeo
between power and simplicity.
Theorem 4.15 (SOSC with strict complementarity). Suppose that x

R
n
is such that
(i) KKT conditions (see Theorem 4.11) hold with

as multipliers, and
j
> 0 j
J(x

).
(ii) h
T

2
L(x

)h > 0 h h ,= 0 :
g
x
(x

)h = 0, f
j
(x

)
T
h = 0 j J(x

)
Then x

is a strict local minimizer.


Proof. See [18].
Remark 4.16 Without strict complementarity, the result is not valid anymore. (Example:
minx
2
y
2
: y 0 with (x

, y

) = (0, 0) which is a KKT point but not a local minimizer.)


An alternative (stronger) condition is obtained by dropping the strict complementarity as-
sumption but replacing in (ii) J(x

) with its subset I(x

) := j :

j
> 0, the set of indices
of binding constraints. Notice that if

j
= 0, the corresponding constraint does not enter
the KKT conditions. Hence, if such non-binding constraints is removed, x

will still be a
KKT point.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 93
Constrained Optimization
Exercise 4.24 Show that the second order suciency condition still holds if condition (iii)
is replaced with
h
T

2
L(x

)h > 0 h S
f,g
(x

)0 .
Find an example where this condition holds while (iii) does not.
This condition is overly strong: see example in Remark 4.15.
Remark 4.17 If spf
j
(x

) : j I(x

) = R
n
, then condition (iii) in the sucient
condition holds trivially, yielding a rst order suciency condition. (Relate this to Exercise
4.20.)
4.7 Glance at Numerical Methods for Constrained
Problems
Penalty functions methods
(P) minf
0
(x) : f(x) 0, g(x) = 0
The idea is to replace (P) by a sequence of unconstrained problems
(P
i
) min
i
(x) f
0
(x) + c
i
P(x)
with P(x) = [[g(x)[[
2
+[[f(x)
+
[[
2
, (f(x)
+
)
j
= max(0, f
j
(x)), and where c
i
grows to innity.
Note that P(x) = 0 i x (feasible set). The rationale behind the method is that, if c
i
is large, the penalty term P(x) will tend to push the solution x
i
(P
i
) towards the feasible
set. The norm used in dening P(x) is arbitrary, although the Euclidean norm has clear
computational advantages (P(x) continuously dierentiable: this is the reason for squaring
the norms).
Exercise 4.25 Show that if [[ [[ is any norm in R
n
, [[ [[ is not continuously dierentiable
at 0.
First, let us suppose that each P
i
can be solved for a global minimizer x
i
(conceptual version).
Theorem 4.16 Suppose x
i
K
x, then x solves (P).
Exercise 4.26 Prove Theorem 4.16
As pointed out earlier, the above algorithm is purely conceptual since it requires exact
computation of a global minimizer for each i. However, using one of the algorithms previously
studied, given
i
> 0, one can construct a point x
i
satisfying
[[
i
(x
i
)[[
i
, (4.47)
by constructing a sequence x
j
such that
i
(x
j
) 0 as j and stopping computation
when (4.47) holds. We choose
i
such that
i
0 as i .
For simplicity, we consider now problems with equality constraints only.
94 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
4.7 Glance at Numerical Methods for Constrained Problems
Exercise 4.27 Show that
i
(x) = f(x) + c
i
g
x
(x)
T
g(x)
Theorem 4.17 Suppose that x R
n
,
g
x
(x) has full row rank. Suppose that x
i
K
x

. Then
x

and


f(x

) +
g
x
(x

)
T

= 0 (4.48)
i.e., the rst order necessary condition of optimality holds at x

. Moreover, c
i
g(x
i
)
K

.
Exercise 4.28 Prove Theorem 4.17
Remark 4.18 The main drawback of this penalty function algorithm is the need to drive
c
i
to in order to achieve convergence to a solution. When c
i
is large (P
i
) is very dicult
to solve (slow convergence and numerical diculties due to ill-conditioning). In practice,
one should compute x
i
for a few values of c
i
, set
i
= 1/c
i
and dene x
i
= x(
i
), and then
extrapolate for
i
= 0 (i.e., c
i
= ). Another approach is to modify (P
i
) as follows, yielding
the method of multipliers, due to Hestenes and Powell.
(P
i
) min f(x) +
1
2
c
i
[[g(x)[[
2
+
T
i
g(x) (4.49)
where

i+1
=
i
+ c
i
g(x
i
) (4.50)
with x
i
the solution to (P
i
).
It can be shown that convergence to the solution x

can now be achieved without having to


drive c
i
to , but merely by driving it above a certain threshold c (see [3] for details.)
Methods of feasible directions (inequality constraints)
For unconstrained problems, the value of the function f at a point x indicates whether x
is stationary and, if not, in some sense how far it is from a stationary point. In constrained
optimization, a similar role is played by optimality functions. Our rst optimality function
is dened as follows. For x R
n

1
(x) = min
hS
max
_
f
j
(x)
T
h, j J
0
(x)
_
(4.51)
with S

= h : [[h[[ 1. Any norm could be used but we will focus essentially on the
Euclidean norm, which does not favor any direction. Since the max is taken over a nite
set (thus a compact set) it is continuous in h. Since S is compact, the minimum is achieved.
Proposition 4.8 For all x R
n
,
1
(x) 0. Moreover, if x ,
1
(x) = 0 if and only if x
is a Fritz John point.
Exercise 4.29 Prove Proposition 4.8
Copyright c _19932011, Andre L. Tits. All Rights Reserved 95
Constrained Optimization
We thus have an optimality function through which we can identify Fritz John points. Now
suppose that
1
(x) < 0 (hence x is not a Fritz John point) and let

h be a minimizer in (4.51).
Then f
j
(x)
T

h < 0 for all j J


0
(x), i.e.,

h is a descent direction for the cost function and
all the active constraints, i.e., a feasible descent direction. A major drawback of
1
(x) is its
lack of continuity, due to jump in the set J
0
(x) when x hits a constraint boundary. Hence
[
1
(x)[ may be large even if x is very close to a Fritz John point. This drawback is avoided
by the following optimality function.

2
(x) = min
hS
max
_
f
0
(x)
T
h; f
j
(x) +f
j
(x),
T
h, j = 1, . . . , m
_
(4.52)
Exercise 4.30 Show that
2
(x) is continuous.
Proposition 4.9 Suppose x . Then
2
(x) 0 and, moreover,
2
(x) = 0 if and only if x
is a Fritz John point.
Exercise 4.31 Prove Proposition 4.9
Hence
2
has the same properties as
1
but, furthermore, it is continuous. A drawback of
2
,
however, is that its computation requires evaluation of the gradients of all the constraints,
as opposed to just those of the active constraints for
1
.
We will see later that computing
1
or
2
, as well as the minimizing h, amounts to solving a
quadratic program, and this can be done quite eciently. We now use
2
(x) in the following
optimization algorithm, which belongs to the class of methods of feasible directions.
Algorithm (method of feasible directions)
Parameters , (0, 1)
Data x
0

i = 0
while
2
(x
i
) ,= 0 do
obtain h
i
= h(x
i
), minimizer in (4.52)
k = 0
repeat
if (f
0
(x
i
+
k
h
i
) f(x
i
)
k
f(x
i
)
T
h & f
j
(x
i
+
k
h
i
) 0 for j = 1, 2, . . . , m)
then break
k = k + 1

forever
x
i+1
= x
i
+
k
h
i
i = i + 1

stop
We state without proof a corresponding convergence theorem (the proof is essentially pat-
terned after the corresponding proof in the unconstrained case).
Theorem 4.18 Suppose that x
i
is constructed by the above algorithm. Then x
i
i
and x
i
K
x implies
2
( x) = 0 (i.e. x is a Fritz John point).
96 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
4.7 Glance at Numerical Methods for Constrained Problems
Note: The proof of this theorem relies crucially on the continuity of
2
(x).
Newtons method for constrained problems
Consider again the problem
minf(x) : g(x) = 0 (4.53)
We know that, if x

is a local minimizer for (4.53) and


g
x
(x

) has full rank, then R


m
such that
_

x
L(x

) 0
g(x

) = 0
which is a system of n+m equations with n+m unknowns. We can try to solve this system
using Newtons method. Dene z = (x, ) and
F(z) =
_

x
L(x, )
g(x)
_
.
We have seen that, in order for Newtons method to converge locally (and quadratically),
it is sucient that
F
z
(z

) be non singular, with z

= (x

), and that
F
z
be Lipschitz
continuous around z

.
Exercise 4.32 Suppose that 2nd order suciency conditions of optimality are satised at
x

and, furthermore, suppose that


g
x
(x

) has full rank. Then


F
z
(z

) is non singular.
As in the unconstrained case, it will be necessary to stabilize the algorithm in order to
achieve convergence from any initial guess. Again such stabilization can be achieved by
using a suitable step-size rule
using a composite algorithm.
Sequential quadratic programming
This is an extension of Newtons method to the problem
minf
0
(x) : g(x) = 0, f(x) 0 . (P)
Starting from an estimate (x
i
,
i
,
i
) of a KKT triple, solve the following minimization
problem:
min
v
L(x
i
,
i
,
i
)
T
v +
1
2
v
T

2
L(x
i
,
i
,
i
)v :
g(x
i
) +
g
x
(x
i
)v = 0, f(x
i
) +
f
x
(x
i
)v 0 (P
i
)
i.e., the constraints are linearized and the cost is quadratic (but is an approximation to L
rather than to f
0
). (P
i
) is a quadratic program (we will discuss those later) since the cost
Copyright c _19932011, Andre L. Tits. All Rights Reserved 97
Constrained Optimization
is quadratic (in v) and the constraints linear (in v). It can be solved exactly and eciently.
Let us denote by v
i
its solution and
i
,
i
the associated multipliers. The next iterate is
x
i+1
= x
i
+ v
i

i+1
=
i

i+1
=
i
Then (P
i+1
) is solved as so on. Under suitable conditions (including second order suciency
condition at x

with multipliers

), the algorithm converges locally quadratically if


(x
i
,
i
,
i
) is close enough to (x

). As previously,

2
L
x
2
can be replaced by an estimate,
e.g., using an update formula. It is advantageous to keep those estimates positive denite. To
stabilize the methods (i.e., to obtain global convergence) suitable step-size rules are available.
Exercise 4.33 Show that, if m = 0 (no inequality) the iteration above is identical to the
Newton iteration considered earlier.
4.8 Sensitivity
An important question for practical applications is to know what would be the eect of
slightly modifying the constraints, i.e., of solving the problem
minf
0
(x) : g(x) = b
1
, f(x) b
2
(4.54)
with the components of b
1
and b
2
being small, i.e., to know how sensitive the result is to a
variation of the constraints. For simplicity, we rst consider the case with equalities only.
Theorem 4.19 Consider the family of problems
minf
0
(x) : g(x) = b (4.55)
where f
0
and g are twice continuously dierentiable. Suppose that for b = 0 there is a local
solution x

such that
g
x
(x

) has full row rank and that, together with its associated multiplier

, it satises the 2nd order suciency condition of optimality. Then there is > 0 s.t.
b B(0, ) x(b), x() continuously dierentiable, such that x(0) = x

and x(b) is a strict


local minimizer for (4.55). Furthermore

b
f
0
(x(0)) =

. (4.56)
Proof. Consider the system
f
0
(x) +
g
x
(x)
T
= 0
g(x) b = 0
_
(4.57)
By hypothesis there is a solution x

to (4.57) when b = 0. Consider now the left-hand


side of (4.57) as a function F(x, , b) of x, and b and try to solve it locally for x and
using IFT (assuming f
0
and g are twice continuously Frechet dierentiable)
F

_
x

_
(x

, 0) =
_

2
L(x

)
g
x
(x

)
T
g
x
(x

) 0
_
(4.58)
98 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
4.8 Sensitivity
which was shown to be non-singular, in Exercise 4.32, under the same hypotheses. Hence
x(b) and (b) are well dened and continuously dierentiable for [[b[[ small enough and
they form a KKT pair for (4.55) since they satisfy (4.57). Furthermore, by continuity, they
still satisfy the 2nd order suciency conditions (check this) and hence, x(b) is a strict local
minimizer for (4.55). By the chain rule, we have

b
f
0
(x(b))

b=0
=
f
0
x
(x

)
x(b)
b

b=0
=

T g
x
(x

)
x(b)
b

b=0
.
But by the second equality in (4.57)
g
x
(x(b))
x(b)
b
=

b
b = I
hence
b
f
0
(x(b))

b=0
=

.
We now state the corresponding theorem for the general problem (4.54).
Theorem 4.20 Consider the family of problems (4.54) where f
0
, f and g are twice con-
tinuously dierentiable. Suppose that for b = (b
1
, b
2
) = 0, there is a local solution x

such
that s

= g
j
(x

), j = 1, 2, . . . , f
j
(x

), j J(x

) is a linearly independent set of


vectors. Suppose that, together with the multipliers

R
n
and

, x

satises SOSC
with strict complementarity. Then there exists > 0 such that for all b B(0, ) there exists
x(b), x() continuously dierentiable, such that x(0) = x

and x(b) is a strict local minimizer


for (4.54). Furthermore

b
1
f
0
(x(0)) =


b
2
f
0
(x(0)) =

The idea of the proof is similar to the equality case. Strict complementarity is needed in
order for (b) to still satisfy (b) 0.
Exercise 4.34 In theorem above, instead of considering (4.58), one has to consider
F

_
_
_
x

_
_
_
(x

, 0) =
_

2
L(x

)
g
x
(x

)
T
f
1
(x

) f
k
(x

)
g
x
(x

)
f
1
(x

)
T
0
.
.
.
f
k
(x

)
T
_

_
where, w.l.o.g., it has been assumed that the active constraints are f
1
, f
2
, . . ., f
k
.
Show that, under the assumption of the theorem above,
F
x
(x

, 0) is non-singular.
(Hence, again, one can solve locally for x(b), (b) and
j
(b) for j J(x

).)
Copyright c _19932011, Andre L. Tits. All Rights Reserved 99
Constrained Optimization
Remark 4.19 Equilibrium price interpretation. Consider, for simplicity, the problem
minf
0
(x) : f(x) 0 with f : R R (1 constraint only)
(the following can easily be extended to the general case). Suppose that, at the expense of
paying a price of p per unit, we can replace the inequality constraint used above by the less
stringent
f(x) b , b > 0
(conversely, we gain p per unit if b < 0), for a total additional price of pb. From the theorem
above, the resulting savings will be, to the rst order
f
0
(x(b)) f
0
(x(0))
d
db
f(x(0))b =

b
i.e., savings of

b. Hence, if p <

, it is to our advantage to relax the constraint by


increasing the right-hand side. If p >

, to the contrary, we can save by tightening the


constraint. If p =

, neither relaxation nor tightening yields any gain (to rst order).
Hence

is called the equilibrium price.


Note. As seen earlier, the linear independence condition in the theorem above (linear
independence of the gradients of the active constraints) insures uniqueness of the KKT
multipliers

and

. (Uniqueness is obviously required for the interpretation of

as
sensitivity.)
Exercise 4.35 Discuss the more general case
minf
0
(x) : g(x, b
1
) = 0, f(x, b
2
) 0.
4.9 Duality
See [26]. Most results given in this section do not require any dierentiability of the objective
and constraint functions. Also, some functions will take values on the extended real line
(including ). The crucial assumption will be that of convexity. The results are global.
We consider the inequality constrained problem
minf
0
(x) : f(x) 0, x X (P)
with f
0
: R
n
R, f : R
n
R
m
and X is a given subset of R
n
(e.g., X = R
n
). As
before, we dene the Lagrangian function by
L(x, ) = f
0
(x) +
m

j=1

j
f
j
(x)
100 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
4.9 Duality
Exercise 4.36 (sucient condition of optimality; no dierentiability or convexity assumed).
Suppose that (x

) X R
m
is such that

0 and
L(x

, ) L(x

) L(x,

) 0, x X, (4.59)
i.e., (x

) is a saddle point for L. Then x

is a global minimizer for (P). (In particular,


it is feasible for (P).)
We will see that under assumptions of convexity and a certain constraint qualication, the
following converse holds: if x

solves (P) then

0 such that (4.59) holds. As we show


below (Proposition 4.10), the latter is equivalent to the statement that
min
xX
_
sup
0
L(x, )
_
= max
0
_
inf
xX
L(x, )
_
and that the left-hand side achieves its minimum at x

and the right-hand side achieves its


maximum at

. If this holds, strong duality is said to hold. In such case, one could compute
any

which globally maximizes () = inf


x
L(x, ), subject to the simple constraint 0.
Once

is known, (4.59) shows that x

is a minimizer of L(x,

), unconstrained if X = R
n
.
(Note: L(x,

) may have other spurious minimizers, i.e., minimizers for which (4.59) does
not hold; see below.)
Instead of L(x, ), we now consider a more general function F : R
n
R
m
R. First of
all, weak duality always holds.
Lemma 4.2 (Weak duality). Given two sets X and Y and a function F : X Y R,
sup
yY
inf
xX
F(x, y) inf
xX
sup
yY
F(x, y) (4.60)
Proof. We have successively
inf
xX
F(x, y) F(x, y) x X, y Y. (4.61)
Hence, taking sup
yY
on both sides,
sup
yY
inf
xX
F(x, y) sup
yY
F(x, y) x X (4.62)
where now only x is free. Taking inf
xX
on both sides (i.e., in the right-hand side) yields
sup
yY
inf
xX
F(x, y) inf
xX
sup
yY
F(x, y) (4.63)
In the sequel, we will make use of the following result, of independent interest.
Proposition 4.10 Given two sets X and Y and a function F : X Y R, the following
statements are equivalent (under no regularity or convexity assumption)
Copyright c _19932011, Andre L. Tits. All Rights Reserved 101
Constrained Optimization
(i) x

X, y

Y and
F(x

, y) F(x

, y

) F(x, y

) x X, y Y
(ii)
min
xX
sup
yY
F(x, y) = max
yY
inf
xX
F(x, y)
where the common value is nite and the left-hand side is achieved at x

and the right-


hand side is achieved at y

.
Proof. ((ii)(i)) Let = min
x
_
sup
y
F(x, y)
_
= max
y
_
inf
x
F(x, y)
_
and let x

and y

achieve
the min in the rst expression and the max in the second expression, respectively. Then
F(x

, y) sup
y
F(x

, y) = = inf
x
F(x, y

) F(x, y

) x, y.
Thus
F(x

, y

) F(x

, y

)
and the proof is complete.
((i)(ii))
inf
x
sup
y
F(x, y) sup
y
F(x

, y) = F(x

, y

) = inf
x
F(x, y

) sup
y
inf
x
F(x, y).
By weak duality (see below) it follows that
inf
x
sup
y
F(x, y) = sup
y
F(x

, y) = F(x

, y

) = inf
x
F(x, y

) = sup
y
inf
x
F(x, y).
Further, the rst and fourth equalities show that the inf and sup are achieved at x

and
y

, respectively.
A pair (x

, y

) satisfying the conditions of Proposition 4.10 is referred to as a saddle point.


Exercise 4.37 The set of saddle points of a function F is a Cartesian product, that is, if
(x
1
, y
1
) and (x
2
, y
2
) are saddle points, then (x
1
, y
2
) and (x
2
, y
1
) also are. Further, F takes
the same value at all its saddle points.
Let us apply the above to L(x, ). Thus, consider problem (P) and let Y = R
m
:
0. Let p : R
n
R +, and : R
m
R be given by
p(x) = sup
0
L(x, ) =
_
f
0
(x) if f(x) 0
+ otherwise
() = inf
xX
L(x, ).
102 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
4.9 Duality
Then (P) can be written
minimize p(x) s.t. x X (4.64)
and weak duality implies that
inf
xX
p(x) sup
0
(). (4.65)
Denition 4.6 It is said that duality holds (or strong duality holds) if equality holds in
(4.65).
Remark 4.20 Some authors use the phrases strong duality or duality holds to means
that not only there is no duality gap but furthermore the primal inmum and dual supremum
are attained at some x

X and

Y .
If duality holds and x

X minimizes p(x) over X and

solves the dual problem


maximize () s.t. 0, (4.66)
it follows that
L(x

, ) L(x

) L(x,

) x X, 0 (4.67)
and, in particular, x

is a global minimizer for


minimize L(x,

) s.t. x X.
Remark 4.21 That some x X minimizes L(x,

) over X is not sucient for x to solve


(P) (even if (P) does have a global minimizer). [Similarly, it is not sucient that 0
maximize L(x

, ) in order for to solve (4.66); in fact it is immediately clear that


maximizing L(x

, ) implies nothing about


j
for j J(x

).] However, as we saw earlier,


( x, ) satisfying both inequalities in (4.67) is enough for x to solve (P) and to solve (4.66).
Suppose now that duality holds, i.e.,
min
xX
sup
0
L(x, ) = max
0
inf
xX
L(x, ) (4.68)
(with the min and the max being achieved). Suppose we can easily compute a maximizer

for the right-hand side. Then, by Proposition 4.10 and Exercise 4.36, there exists x

X
such that (4.59) holds, and such x

is a global minimizer for (P). From (4.59) such x

is
among the minimizers of L(x,

). The key is thus whether (4.68) holds. This is known as


(strong) duality. We rst state without proof a more general result, about the existence of
a saddle point for convex-concave functions. (This and other related results can be found
in [5]. The present result is a minor restatement of Proposition 2.6.4 in that book.)
Copyright c _19932011, Andre L. Tits. All Rights Reserved 103
Constrained Optimization
Theorem 4.21 Let X and Y be convex sets and let F : X Y R be convex in its rst
argument and concave (i.e., F is convex) in its second argument, and further suppose that,
for each x X and y Y , the epigraphs of F(, y) and F(x, ) are closed. Further suppose
that the set of minimizers of sup
yY
F(x, y) is nonempty and compact. Then
min
xX
sup
yY
F(x, y) = max
yY
inf
xX
F(x, y).
We now prove this result in the specic case of our Lagrangian function L.
Proposition 4.11 ()

= inf
x
L(x, ) is concave (i.e., is convex) (without convexity
assumption)
Proof. L(x, ) is ane, hence concave, and the pointwise inmum of a set of concave functions
is concave (prove it!).
We will now see that convexity of f
0
and f and a certain stability assumption (related to
KTCQ) are sucient for duality to hold. This result, as well as the results derived so far in
this section, holds without dierentiability assumption. Nevertheless, we will rst prove the
dierentiable case with the additional assumption that X = R
n
. Indeed, in that case, the
proof is immediate.
Theorem 4.22 Suppose x

solves (P) with X = R


n
, suppose that f
)
and f are dierentiable
and that KTCQ holds, and let

be a corresponding KKT multiplier vector. Furthermore,


suppose f
0
and f are convex functions. Then

solves the dual problem, duality holds and


x

minimizes L(x,

).
Proof. Since (x

) is a KKT pair one has


f
0
(x

) +
m

j=1

j
f
j
(x

) = 0 (4.69)
and by complementary slackness
L(x

) = f
0
(x

) = p(x

).
Now, under our convexity assumption,
(x) := f
0
(x) +
m

j=1

j
f
j
(x) = L(x,

) (4.70)
is convex, and it follows that x

is a global minimum for L(x,

). Hence (

) = L(x

).
It follows that

solves the dual, duality holds, and, from (4.69), x

solves

x
L(x

) = 0
104 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
4.9 Duality
Remark 4.22 The condition above is also necessary for

to solve the dual: indeed if is


not a KKT multiplier vector at x

then
x
L(x

, ) ,= 0 so that x

does not minimize L(, ),


and (x

, ) is not a saddle point.


Remark 4.23
1. The only convexity assumption we used is convexity of (x) = L(x,

), which is weaker
than convexity of f
0
and f (for one thing, the inactive constraints are irrelevant).
2. A weaker result, called local duality is as follows: If the mins in (4.64)-(4.66) are taken
in a local sense, around a KKT point x

, then duality holds with only local convexity


of L(x,

). Now, if 2nd order suciency conditions hold at x

, then we know that

2
x
L(x

) is positive denite over the tangent space to the active constraints. If


positive deniteness can be extended over the whole space, then local (strong) convexity
would hold. This is in fact one of the ideas which led to the method of multipliers
mentioned above (also called augmented Lagrangian).
Exercise 4.38 Dene again (see Method of Feasible Directions in section 4.7)
(x) = min
[[h[[
2
1
maxf
j
(x)
T
h : j J
0
(x). (4.71)
Using duality, show that
(x) = min[[

jJ
0
(x)

j
f
j
(x)[[
2

jJ
0
(x)

j
= 1,
j
0 j J
0
(x) (4.72)
and if h

solves (4.71) and

solves (4.72) then, if (x) ,= 0, we have.


h

jJ
0
(x)

j
f
j
(x)
[[

jJ
0
(x)

j
f
j
(x)[[
2
. (4.73)
Hint: rst show that the given problem is equivalent to the minimization over
_
h

h
_
R
n+1
min
h

h[f
j
(x)
T
h

h j J
0
(x), h
T
h 1 .
Remark 4.24 This shows that the search direction corresponding to is the direction
opposite to the nearest point to the origin in the convex hull of the gradients of the active
constraints. Notice that applying duality has resulted in a problem (4.72) with fewer variables
and simpler constraints than the original problem (4.71). Also, (4.72) is a quadratic program
(see below).
We now drop the dierentiability assumption on f
0
and f and merely assume that they are
convex. We substitute for (P) the family of problems
minf
0
(x) : f(x) b, x X
Copyright c _19932011, Andre L. Tits. All Rights Reserved 105
Constrained Optimization

,f(x)
f (x)
0
slope=-0
L(x,)

(f(x),f (x))
0
(xX)
z =f (x)-,z-f(x)=L(x,)-,z 0 0

m
z

z0

Figure 4.9:
with b R
m
and X a convex set, and we will be mainly interested in b in a neighborhood
of the origin. We know that, when f
0
and f are continuously dierentiable, the KKT
multipliers can be interpreted as sensitivities of the optimal cost to variation of components
of f. We will see here how this can be generalized to the non-dierentiable case. When the
generalization holds, duality will hold.
The remainder of our analysis will take place in R
m+1
where points of the form
(f(x), f
0
(x)) lie (see Figure 4.9). We will denote vectors in R
m+1
by (z, z
0
) where z
0
R,
z R
m
. We also dene

f : X R
m+1
by

f(x) =
_
f(x)
f
0
(x)
_
and

f(X) by

f(X) =

f(x) : x X
On Figure 4.9, the cross indicates the position of

f(x) for some x X; and, for some 0,
the oblique line represents the hyperplane H
x,
orthogonal to (, 1) R
m+1
, i.e.,
H
x,
= (z, z
0
) : (, 1)
T
(z, z
0
) = ,
where is such that

f(x) H
x,
, i.e.,

T
f(x) + f
0
(x) = ,
i.e.,
z
0
= L(x, )
T
z.
In particular, the oblique line intersects the vertical axis at z
0
= L(x, ).
Next, we dene the following objects:
(b) = x X : f(x) b,
B = b R
m
: (b) ,= ,
V : R
m
R , with V (z) := inf
xX
f
0
(x) : f(x) z.
V is the value function. It can take the values +(when x X : f(x) b is empty) and
(when V is unbounded from below on x X : f(x) b).
106 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
4.9 Duality
Exercise 4.39 If f
0
, f are convex and X is a convex set, then (b) and B and the epigraph
of V are convex sets (so that V is a convex function), and V is monotonic nonincreasing.
(Note that, on the other hand, minf
0
(x) : f(x) = b need not be convex in b: e.g., f(x) = e
x
and f
0
(x) = x. Also,

f(X) need not be convex (e.g., it could be just a curve as, for instance,
if n = m = 1).)
The following exercise points out a simple geometric relationship between epiV and

f(X),
which yields simple intuition for the central result of this section.
Exercise 4.40 Prove that
cl(epiV ) = cl(

f(X) +R
m+1
+
),
where R
m+1
+
is the set of points in R
m+1
with nonnegative components. Further, if for all z
such that V (z) is nite, the min in the denition of V is attained, then
epiV =

f(X) +R
m+1
+
.
This relationship is portrayed in Figure 4.10, which immediately suggests that the lower
tangent plane to epiV with slope (i.e., orthogonal to (, 1), with 0), intersects
the vertical axis at ordinate (), i.e.,
infz
0
+
T
z : (z, z
0
) epiV = inf
xX
L(x, ) = () 0. (4.74)
We now provide a simple, rigorous derivation of this result.
First, the following identity is easily derived.
Exercise 4.41
infz
0
+
T
z : z
0
V (z) = infz
0
+
T
z : z
0
> V (z).
The claim now follows:
infz
0
+
T
z : (z, z
0
) epiV = inf
(z,z
0
)R
m+1
z
0
+
T
z : z
0
V (z)
= inf
(z,z
0
)R
m+1
z
0
+
T
z : z
0
> V (z)
= inf
(z,z
0
)R
m+1
,xX
z
0
+
T
z : z
0
> f
0
(x), f(x) z
= inf
(z,z
0
)R
m+1
,xX
z
0
+
T
f(x)
= inf
xX
f
0
(x) +
T
f(x)
= inf
xX
L(x, )
= (),
where we have used the fact that 0.
In the case of the picture, V (0) = f
0
(x

) = (

) i.e., duality holds and inf and sup are


achieved and nite. This works because
Copyright c _19932011, Andre L. Tits. All Rights Reserved 107
Constrained Optimization
(i) V () is convex
(ii) there exists a non-vertical supporting line (supporting hyperplane) to epi V at (0, V (0)).
(A sucient condition for this is that 0 int B.)
Note that if V () is continuously dierentiable at 0, then

is its slope where

is the
KKT multiplier vector. This is exactly the sensitivity result we proved in section 4.8!
In the convex (not necessarily dierentiable) case, condition (ii) acts as a substitute for
KTCQ. It is known as the Slater constraint qualication. It says that there exists some
feasible x at which none of the constraints is active, i.e., f(x) < 0. Condition (i), which
implies convexity of epi V , implies the existence of a supporting hyperplane, i.e., (,
0
) ,= 0,
s.t.

T
0 +
0
V (0)
T
z +
0
z
0
(z, z
0
) epi V
i.e.

0
(V (0))
0
z
0
+
T
z (z, z
0
) epi V (4.75)
and the epigraph property of epi V ((0, ) epi V > V (0)) implies that
0
0.
Exercise 4.42 Let S be convex and closed and let x S. Then there exists a hyperplane
H separating x and S. H is called supporting hyperplane to S at x. (Note: The result still
holds without the assumption that S is closed, but this is harder to prove. Hint: If S is
convex, then S = clS.)
Exercise 4.43 Suppose f
0
and f
j
are continuously dierentiable and convex, and suppose
Slaters constraint qualication holds. Then MFCQ holds at any feasible x

.
Under condition (ii),
0
> 0, i.e., the supporting hyperplane is non-vertical. In particular
suppose that 0 int B and proceed by contradiction: if
0
= 0 (4.75) reduces to
T
z 0
for all (z, z
0
) epi V , in particular for all z with |z| small enough; since (
0
, ) ,= 0, this
is impossible. Under this condition, dividing through by
0
we obtain:

R
m
s.t. V (0) z
0
+ (

)
T
z (z, z
0
) epi V .
Next, the fact that V is monotonic non-increasing implies that

0. (Indeed we can keep


z
0
xed and let any component of z go to +.) Formalizing the argument made above, we
now obtain, since, for all x X, (f
0
(x), f(x)) epi V ,
f
0
(x

) = V (0) f
0
(x) + (

)
T
f(x) = L(x,

) x X ,
implying that
f
0
(x

) inf
x
L(x,

) = (

) .
In view of weak duality, duality holds and sup
0
() is achieved at

0.
108 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
4.10 Linear programming
Remark 4.25 Figure 4.10 may be misleading, as it focuses on the special case where m = 1
and the constraint is active at the solution (since

> 0). Figure 4.11 illustrates other cases.


It shows V (z) for z = e
j
, with a scalar and e
j
the jth coordinate vector, both in the case
when the constraint is active and in the case it is not.
To summarize: if x

solves (P) (in particular, inf


xX
p(x) is achieved and nite) and
conditions (i) and (ii) hold, then sup
0
() is achieved and equal to f
0
(x

) and x

solves
minimize L(x,

) s.t. x X .
Note. Given a solution

to the dual, there may exist x minimizing L(x,

) s.t. x does not


solve the primal. A simple example is given by the problem
min (x) s.t. x 0
where L(x,

) is constant (V () is a straight line). For, e.g., x = 1,


L( x,

) L(x,

) x
but it is not true that
L( x, ) L( x,

) 0
4.10 Linear programming
(See [18])
Consider the problem
minc
T
x : Gx = b
1
, Fx b
2
(4.76)
where
c R
n
G R
mn
F R
kn
For simplicity, assume that the feasible set is nonempty and bounded. Note that if n = 2
and m = 0, we have a picture such as that on Figure 4.12. Based on this gure, we make
the following guesses
1. If a solution exists (there is one, in view of our assumptions), then there is a solution
on a vertex (there may be a continuum of solutions, along an edge)
2. If all vertices adjacent to x have a larger cost, then x is optimal
Hence a simple algorithm would be
1. Find x
0
, a vertex. Set i = 0
Copyright c _19932011, Andre L. Tits. All Rights Reserved 109
Constrained Optimization
2. If all vertices adjacent to x
i
have higher cost, stop. Else, select an edge along which
the directional derivative of c
T
x is the most negative. Let x
i+1
be the corresponding
adjacent vertex and iterate.
This is the basic idea of simplex algorithm.
In the sequel, we restrict ourselves to the following canonical form
minc
T
x : Ax = b, x 0 (4.77)
where c R
n
, A R
mn
, b R
m
Proposition 4.12 Consider the problem (4.76). Also consider
minc
T
(v w) : G(v w) = b
1
, F(v w) + y = b
2
, v 0, w 0, y 0 (4.78)
if ( v, w, y) solves (4.78) then x = v w solves (4.76).
Exercise 4.44 Prove Proposition 4.2.
Problem (4.78) is actually of the form (4.77) since it can be rewritten as
min
_
_
_
_
c
c
0
_
_
_
T
_
_
_
v
w
y
_
_
_

_
G G 0
F F I
_
_

_
v
w
y
_

_ =
_
b
1
b
2
_
,
_
_
_
v
w
y
_
_
_ 0
_
Hence, we do not lose any generality by considering (4.77) only. To give meaning to our rst
guess, we need to introduce a suitable notion of vertex or extreme point.
Denition 4.7 Let R
n
be a polyhedron (intersection of half spaces). Then x is an
extreme point of if
x = x
1
+ (1 )x
2
(0, 1); x
1
, x
2

_
x = x
1
= x
2
Proposition 4.13 (See [18])
Suppose (4.77) has a solution (it does under our assumptions). Then there exists x

, an
extreme point, such that x

solves (4.77).
Until the recent introduction of new ideas in linear programming (Karmarkar and others),
the only practical method for the solution of general linear programs was the simplex method.
The idea is as follows:
1. obtain an extreme point of , x
0
, set i = 0.
2. if x
i
is not a solution, pick among the components of x
i
which are zero, a component
such that its increase (x
i
remaining on Ax = b) causes a decrease in the cost function.
Increase this component, x
i
remaining on Ax = b, until the boundary of is reached
(i.e., another component of x is about to become negative). The new point, x
i+1
, is an
extreme point adjacent to x
i
with lower cost.
110 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
4.10 Linear programming
Obtaining an initial extreme point of can be done by solving another linear program
min
j
: Ax b = 0, x 0, 0 (4.79)
Exercise 4.45 Let ( x, ) solve (4.79) and suppose that
minc
T
x : Ax = b, x 0
has a feasible point. Then = 0 and x is feasible for the given problem.
More about the simplex method can be found in [18].
Quadratic programming (See [7])
Problems with quadratic cost function and linear constraints frequently appear in optimiza-
tion (although not as frequently as linear programs). We have twice met such problems
when studying algorithms for solving general nonlinear problems: in some optimality func-
tions and in the sequential quadratic programming method. As for linear programs, any
quadratic program can be put into the following canonical form
min
1
2
x
T
Qx + c
T
x : Ax = b, x 0 (4.80)
We assume that Q is positive denite so that (4.80) (with strongly convex cost function and
convex feasible set) admits at most one K.T. point, which is then the global minimizer. K.T.
conditions can be expressed as stated in the following theorem.
Theorem 4.23 x solves (4.80) if and only if R
n
, R
n

A x = b, x 0
Q x + c + A
T
= 0

T
x = 0
0
Exercise 4.46 Prove Theorem 4.23.
In this set of conditions, all relations except for the next to last one (complementary slack-
ness) are linear. They can be solved using techniques analog to the simplex method (Wolfe
algorithm; see [7]).
Copyright c _19932011, Andre L. Tits. All Rights Reserved 111
Constrained Optimization
Figure 4.10:
112 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
4.10 Linear programming
Figure 4.11:
Copyright c _19932011, Andre L. Tits. All Rights Reserved 113
Constrained Optimization

f ,x = b
1 1
2
f ,x = b
2 2
2
f ,x = b
4 4
2
f ,x = b
3 3
2

c
x

c,x = constant
Figure 4.12:
114 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
Chapter 5
Calculus of Variations and Optimal
Control
This chapter deals with a subclass of optimization problems of prime interest to controls
theorists and practitioners, that of optimal control problems, and with the classical eld of
calculus of variations (whose formalisation dates back to the late 18th century), a precursor
to continuous-time optimal control. Optimal control problem can be optimization problems
in nite-dimensional spaces (like in the case of discrete-time optimal control with nite
horizon), or in innite-dimensional cases (like in the case of a broad class of continuous-time
optimal control problems). In the latter case, nite-dimensional optimization ideas often
still play a key role, at least when the underlying state space is nite-dimensional.
5.1 Introduction to the calculus of variations
In this section, we give a brief introduction to the classical calculus of variations, and fol-
lowing the discussion in [25], we show connections with optimal control and Pontryagins
Principle.
Let V := [C
1
([a, b])]
n
, a, b R, let
= x V : x(a) = A, x(b) = B (5.1)
with A, B R
n
given and let
J(x) =
b
_
a
L(x(t), x(t), t)dt (5.2)
where L : R
n
R
n
[a, b] R is a given smooth function. The basic problem in the
classical calculus of variations is
minimize J(x) s.t. x . (5.3)
Note at this point that problem (5.3) can be thought of as the optimal control problem
minimize
b
_
a
L(x(t), x(t), t)dt s.t. x(t) = u(t) t, x , u continuous, (5.4)
Copyright c _19932011, Andre L. Tits. All Rights Reserved 115
Calculus of Variations and Optimal Control
where minimization is to be carried out over the pair (x, u).
It turns out that much can be said about this problem, even about local minimizers,
without need to specify a norm on V . In particular, the tangent cone is norm-independent,
as shown in the following exercise.
Exercise 5.1 Show that, regardless of the norm on V , for any x

,
TC(x

, ) = v V : v(a) = v(b) = 0.
Furthermore, TC(x

, ) is a subspace, i.e., a tangent plane.


Suppose now that x

V is a local minimizer fo problem (5.3), according to some norm. It


is readily proved that, given any v , the function f
v
: R R, J(x

+ v) attains
a local minimum at = 0. Prompted by the connection pointed out above with optimal
control, we denote by
L
u
the derivative of L with respect to its second argument.
Exercise 5.2 Prove the above, and further prove that f
v
is continuously dierentiable with
derivative at = 0 given by
f

v
(0) =
b
_
a
_
L
x
(x

(t), x

(t), t)v(t)dt +
L
u
(x

(t), x

(t), t) v(t)dt
_
.
The following key result follows.
Proposition 5.1 If x

is optimal or if x

is a local minimizer for problem (5.3), then


_
b
a
_
L
x
(x

(t), x

(t), t)v(t) +
L
u
(x

(t), x

(t), t) v(t)
_
dt = 0 (5.5)
v V such that v(a) = v(b) = 0
We now proceed to transform (5.5) to obtain a more manageable condition: the Euler-
Lagrange equation. The following derivation is simple but assumes that x

is twice continu-
ously dierentiable. (An alternative, which does not require such assumption, is tu use the
DuBois-Reymond Lemma: see, e.g., [17].) Integrating (5.5) by parts, one gets
_
b
a
L
x
(x

(t), x

(t), t)v(t)dt+
_
L
u
(x

(t), x

(t), t)v(t)
_
b
a

_
b
a
d
dt
_
L
u
(x

(t), x

(t), t)
_
v(t)dt = 0
v V such that v(a) = v(b) = 0 i.e,
_
b
a
_
L
x
(x

(t), x

(t), t)
d
dt
L
u
(x

(t), x

(t), t)
_
v(t)dt = 0 (5.6)
v V such that v(a) = v(b) = 0.
Since the integrand is continuous, the Euler-Lagrange equation follows
116 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
5.1 Introduction to the calculus of variations
Theorem 5.1 (Euler-Lagrange Equation.) If x

is optimal or if x

is a local minimizer
for problem (5.3), then
L
x
(x

(t), x

(t), t)
d
dt
L
u
(x

(t), x

(t), t) = 0 t [a, b] (5.7)


Indeed, it is a direct consequence of the following result.
Exercise 5.3 If f is continuous and if
_
b
a
f(t)v(t)dt = 0 v V v(a) = v(b) = 0
then
f(t) = 0 t [a, b]
Remark 5.1
1. Euler-Lagrange equation is a necessary condition of optimality. If x

is a local mini-
mizer for (5.3) (in particular, x

C
1
), then it satises E.-L. Problem (5.3) may also
have solutions which are not in C
1
, not satisfying E.-L.
2. E.-L. amounts to a second order equation in x with 2 point boundary conditions
(x

(a) = A, x

(b) = B). Existence and uniqueness of a solution is not guaranteed


in general.
Example 5.1 (see [11] for details)
Among all the curves joining 2 given points (x
1
, t
1
) and (x
2
, t
2
) in R
2
, nd the one which
generates the surface of minimum area when rotated about the t-axis.
The area of the surface of revolution generated by rotating the curve x around the t-axis is
J(x) = 2
_
t
2
t
1
x

1 + x
2
dt (5.8)
The Euler-Lagrange equation can be integrated to give
x

(t) = C cosh
t + C
1
C
(5.9)
where C and C
1
are constants to be determined using the boundary conditions.
Exercise 5.4 Check (5.9).
It can be shown that 3 cases are possible, depending on the positions of (x
1
, t
1
) and (x
2
, t
2
)
1. There are 2 curves of the form (5.9) passing through (x
1
, t
1
) and (x
2
, t
2
) (in limit cases,
these 2 curves are identical). One of them solves the problem (see Figure 5.1).
Copyright c _19932011, Andre L. Tits. All Rights Reserved 117
Calculus of Variations and Optimal Control
(a)
x (x ,t )
1 1
(x ,t )
2 2
t

Figure 5.1:
(b)
surface = sum
of 2 disks


Figure 5.2:
2. There are 2 curves of the form (5.9) passing through (x
1
, t
1
) and (x
2
, t
2
). One of them
is a local minimizer. The global minimizer is as in (3) below (non smooth).
3. There is no curve of the form (5.9) passing through (x
1
, t
1
) and (x
2
, t
2
). Then there is
no smooth curve that achieves the minimum. The solution is not C

and is shown in
Figure 5.2 below (it is even not continuous).
Various extensions
The following classes of problems can be handled in a similar way as above.
(1) Variable end points
x(a) and x(b) may not be specied, as well as either a or b (free time problems)
some of the above may be constrained without being xed, e.g.,
g(x(a)) 0
(2) Isoperimetric problems: One can have constraints of the form
K(x)

=
_
b
a
G(x(t), x(t), t)dt = given constant
For more detail, see [17, 11].
Theorem 5.2 (Legendre second-order condition.) If x

is optimal or if x

is a local
minimizer for problem (5.3), then

2
L
u
2
(x

(t), x

(t), t) _ 0 t [a, b].


118 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
5.1 Introduction to the calculus of variations
Again, see [17, 11] for details.
Toward Pontryagins Principle (see [25])
Dene pre-Hamiltonian H : R
n
R
n
R
n
R R (which generalizes the one dened
in section!2.1.1) by
H(x, u, p, t) := p
T
u + L(x, u, t) (5.10)
and, given a (local or global) minimizer x

(t) to problem (5.3), let


p

(t) :=
u
L(x

(t), x

(t), t) t.
Then, from the denition of H, we obtain

u
H(x

(t), x

(t), p

(t), t) = p

(t) p

(t) = 0 t. (5.11)
Also, since
p
H(x, u, p, t) = u, we have

p
H(x

(t) x

(t), p

(t), t) = x

(t) t. (5.12)
Further, the Euler-Lagrange equation yields
p

(t) =
x
L(x

(t), x

(t), p

(t), t) = H(x

(t), x

(t), p

(t), t) t. (5.13)
Finally, Legendres second-order condition yields

2
u
H(x

(t), x

(t), p

(t), t) _ 0 t. (5.14)
Equations (5.11)-(5.12)-(5.13)-(5.14), taken together, are very close to Pontryagins Prin-
ciple applied to problem (5.4). The only missing piece is that, instead of (recall that
x

(t) = u

(t))
H(x

(t), u

(t), p

(t), t) = min
vR
n
H(x

(t), v, p

(t), t), (5.15)


we merely have (5.11) and (5.14), which are necessary conditions for u

(t) to be such min-


imizer. Also note that formulation (5.4) is more general than it may look at rst glance:
in particular, if the given dynamics is of the form x(t) = f(x(t), u(t)) and the equation
v = f(x, u) can be solved for u, then by making a change of variables from u to v, we
can recast the problem into form (5.11). The main missing component in (5.4) is pointwise
constraints on the values of u, say u(t) U for all t, with U a given subset of R
n
, which
is crucial to modern optimal control. With equations (5.11)-(5.12)-(5.13)-(5.14) in hand
though, it is tempting to conjecture that, when such constraints are present, one need only
replace in (5.15) the minimization over R
n
with a minimization over U. This intuition turns
out to be correct indeed, as we will see below (and as we have already seen, in a limited
context, in section 2.2).
Connection with classical mechanics
Classical mechanics extensively refers to a Hamiltonian which is the total energy in
the system. This quantity can be linked to the above as follows. (See [8, Section 1.4] for
additional insight.)
Copyright c _19932011, Andre L. Tits. All Rights Reserved 119
Calculus of Variations and Optimal Control
Consider an isolated mechanical system and denote by x(t) the vector of its position
(and angle) variables. According to the Principle of Least Action (which should be more
appropriately called Principle of Stationary Action), the state of such system evolves so as
to annihilate the derivative of the action o(x), where o : C[t
0
, t
f
] R is given by
o(x) =
_
t
f
t
0
/(x(t), x(t), t)dt,
where
/(x(t), x(t), t) = T V,
T and V being the kinetic and potential energies. As above, with the dynamics x(t) = u(t)
in mind, consider the pre-Hamiltonian
H(x, v, p, t) := p
T
v /(x, v, t). (5.16)
Here the sign change (substitution of / for L) as compared to (5.10) is immaterial since
we are merely seeking a stationary point of o. Mere stationarity of the objective function
does not guarantee that (5.15) will hold, but rst order condition (5.11) was derived based
on stationary only, so it still holds. With (5.16),
H
v
(u) = 0 yields
p =
/
v
(x, u, t). (5.17)
In classical mechanics, the potential energy does not depend on x(t), while the kinetic
energy is of the form T =
1
2
x(t)Mx(t), where M is a symmetric, positive denite matrix.
Substituting into (5.17) yields p = Mu. Invoking the dynamics x(t) = u(t) we then get
(with p(t) = M x(t))
H(x(t), x(t), p(t), t) = x(t)M x(t) T + V = T + V,
i.e., the pre-Hamiltonian evaluated along the trajectory that makes the action stationary is
indeed the total energy in the system.
5.2 Discrete-Time Optimal Control
(see [26])
Consider the problem (time varying system)
min (x
N
) s.t. (5.18)
x
i+1
= x
i
+ f(x
i
, u
i
, i), i = 0, 1, . . . , N 1 (dynamics)
1
g
0
(x
0
) = 0, h
0
(x
0
) 0 (initial state constraints)
g
N
(x
N
) = 0, h
N
(x
N
) 0 (nal state constraints)
q
i
(u
i
) 0 i = 0, . . . , N 1 (control constraints)
where all functions are continuously dierentiable in the xs and us and u
i
R
m
, x
i
R
n
,
: R
n
R and all other functions are real vector-valued.
1
or equivalently, x
i

= x
i+1
x
i
= f(x
i
, u
i
, i)
120 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
5.2 Discrete-Time Optimal Control
Remark 5.2
1. To keep things simple, we will not consider trajectory constraints of the type
r(x
i
, u
i
, i) 0 .
2. Oftentimes the objective is of the form
min
N1

i=0
f
0
(x
i
, u
i
, i) +

(x
N
)
where f
0
is continuously dierentiable in x and u and has values in R. This can be
transcribed into the form (5.18) by adding an extra state, x
0
i
R
1
, i = 0, . . . , N 1
with
x
0
i+1
= x
0
i
+ f
0
(x
i
, u
i
, i) x
0
0
= 0
added to the dynamics and
(x
N
, x
0
N
) = x
0
N
+

(x
N
)
as objective function.
The given problem can be formulated as
min

f
0
(z) :

f(z) 0, g(z) = 0 (5.19)
where
z =
_

_
x
0
.
.
.
x
N
u
0
.
.
.
u
N1
_

_
R
(N+1)n+Nm
is an augmented vector on which to optimize and

f
0
(z) = (x
N
)

f(z) =
_

_
q
0
(u
0
)
.
.
.
q
N1
(u
N1
)
h
0
(x
0
)
h
N
(x
N
)
_

_
g(z) =
_

_
x
1
+ x
0
+ f(x
0
, u
0
, 0)
.
.
.
x
N
+ x
N1
+ f(x
N1
, u
N1
, N 1)
g
0
(x
0
)
g
N
(x
N
)
_

_
(the dynamics are now handled as constraints). If z is optimal, the Fritz John conditions
hold for (5.19).
Copyright c _19932011, Andre L. Tits. All Rights Reserved 121
Calculus of Variations and Optimal Control
i.e.,
(
0
..

,
0
, . . . ,
N1
. .
q
,
0
,
N
. .
h
) 0, p
1
, . . . , p
N
. .
dyn.
,
0
,
N
. .
g
, not all zero s.t.

x
0
p
1
+
f
x
( x
0
, u
0
, 0)
T
p
1
+
g
0
x
( x
0
)
T

0
+
h
0
x
( x
0
)
T

0
. .

=p
0
()
= 0 (5.20)

x
i
p
i
+ p
i+1
+
f
x
( x
i
, u
i
, i)
T
p
i+1
= 0 i = 1, . . . , N 1 (5.21)

x
N

0
( x
N
) p
N
+
g
N
x
( x
N
)
T

N
+
h
N
x
( x
N
)
T

N
= 0 (5.22)

u
i


u
f( x
i
, u
i
, i)
T
p
i+1
+

u
q
i
( u
i
)
T

i
= 0 i = 0, . . . , N 1 (5.23)
+ complementarity slackness

T
i
q
i
( u
i
) = 0 i = 0, . . . , N 1 (5.24)

T
0
h
0
( x
0
) = 0
T
N
h
N
( x
N
) = 0 (5.25)
To simplify these conditions, let us dene the pre-Hamiltonian function H : R
n
R
m
R
n

0, 1, . . . , N 1 R
H(x, u, p, i) = p
T
f(x, u, i)
Then we obtain
(5.20) + (5.21) + () p
i
= p
i+1
+
x
H( x
i
, u
i
, p
i+1
, i) i = 0, 1, . . . , N 1 (5.26)
(5.23) + (5.24)
_

u
H( x
i
, u
i
, p
i+1
, i)
T
+

u
q
i
( u
i
)
T

i
= 0

T
i
q
i
( u
i
) = 0
_
i = 0, . . . , N 1 (5.27)
and (5.27) is a necessary condition of optimality (assuming KTCQ holds) for the problem
min
u
H( x
i
, u, p
i+1
, i) s.t. q
i
(u) 0, i = 0, . . . , N 1
to have u
i
as a solution. So, this is a weak version of Pontryagins Principle.
Also, (5.22)+(5.23)+(5.25)+() imply
transversality conditions
_

_
p
0
tangent plane to active constraints among the
h
j
0
+ equality constraints g
0
at x
0
p
N

0
(x
N
) tangent plane to active constraints
among the h
j
N
+ equality constraints g
N
at x
N
(Note that the later implies that
p
N
A
_

_
/x
g/x
h

/x
_

_ , (5.28)
but (5.28) does not capture the fact that
0
,= 0 if all other multipliers vanish.)
122 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
5.3 Continuous-Time Optimal Control - Linear Systems
Remark 5.3
1. If KTCQ holds for the original problem, one can set
0
= 1. Then if there are no
constraints on x
N
(no g
N
s or h
N
s) we obtain that p
N
= (x
N
).
2. If
0
,= 0, then without loss of generality we can assume
0
= 1 (this amounts to
scaling p).
3. If x
0
is xed, p
0
is free (i.e, no additional information is known about p
0
). Every degree
of freedom given to x
0
results in one constraint on p
0
. If
0
,= 0 (i.e., wlog,
0
= 1),
the same is true for x
N
and p
N
.
4. If all equality constraints (including dynamics) are ane and the objective function
and inequality constraints are convex, then (5.27) is also a sucient condition for the
problem
min
u
H( x
i
, u, p
i+1
, i) s.t. q
i
(u) 0, i = 1, . . . , N 1 (5.29)
to have a local minimum at u
i
, i.e., a true Pontryagin Principle holds: there exist
vectors p
0
, p
1
, . . . , p
N
satisfying (5.26) (dynamics) such that u
i
solves the constraint
optimization problem (5.29) for i = 1, . . . , N, and such that the transversality condi-
tions hold. The vector p
i
is known as the co-state or adjoint variable (or dual variable)
at time i. Equation (5.26) is the adjoint dierence equation. Note that problems (5.29)
are decoupled in the sense that the kth problem can be solved for u
k
as a function of x
k
and p
k+1
only. We will discuss this further in the context of continuous-time optimal
control.
Exercise 5.5 Consider the objective
min
N1

i=0
f
0
(x
i
, u
i
, i)
Dene H(x, u, p, i) = p
0
f
0
(x, u, i) + p
T
f(x, u, i)
with p =
_
p
0
p
_
scalar
Show that relationships (5.26)-(5.27) hold with this new H, and p
0
0, p
0
constant.
5.3 Continuous-Time Optimal Control - Linear Sys-
tems
See [26]
We rst briey review the relevant results obtained in section 2.2. (See that section for
denitions and notations.)
Copyright c _19932011, Andre L. Tits. All Rights Reserved 123
Calculus of Variations and Optimal Control
Consider the linear system (time varying)
(S) x(t) = A(t)x(t) + B(t)u(t), t t
0
where A(), B() are assumed to be piecewise continuous (nitely many discontinuities on
any nite interval, with nite left and right limits).
Let c R
n
, c ,= 0, x
0
R
n
and let t
f
t
0
be a xed time. We are concerned with the
following problem. Find a control u

| so as to
minimize c
T
x(t
f
) s.t.
dynamics x(t) = A(t)x(t) + B(t)u(t) t
0
t t
f
initial condition x(t
0
) = x
0
nal condition x(t
f
) R
n
(no constraints)
control constraint u |
The following result was established in section 2.2. Let u

| and let
x

(t) = (t, t
0
, x
0
, u

), t
0
t t
f
.
Let p

(t) satisfy the adjoint equation:


p

(t) = A
T
(t)p

(t) t
0
t t
f
with nal condition
p

(t
f
) = c
Then u

is optimal if and only if


p

(t)
T
B(t)u

(t) = infp

(t)
T
B(t)v : v U (5.30)
(implying that the inf is achieved) for all t [t
0
, t
f
]. [Note that the minimization is now
over a nite dimension space.] Further, condition (2.22) can be equivalently expressed in
terms of the pre-Hamiltonian H(, , v, ) =
T
(A() +B()v) and associated Hamiltonian
1 as
H(t, x

(t), u

(t), p

(t)) = 1(t, x

(t), p

(t)) t (5.31)
Geometric interpretation
Denition 5.1 Let K R
n
, x

K. We say that d is the inward normal to a hyperplane


supporting K at x

if d ,= 0 and
d
T
x

d
T
x x K
Remark 5.4 Equivalently
d
T
(x x

) 0 x K
i.e., from x

, all the directions towards a point in K make with d an angle of less than 90

.
124 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
5.3 Continuous-Time Optimal Control - Linear Systems
t
0
t
1
t
2
t
f
0
0 2
K(t ,t ,x )
p (t ) *
0
0
0 f
K(t ,t ,x )
p (t ) *
1
p (t ) *
2
p (t )=c *
f
R
n
x (t ) *
2
x (t ) *
f
0 1
K(t ,t ,x )
0

x = x (t )
= K(t ,t ,x )
0 0
0
0 *
0
1
x (t ) *


Figure 5.3:
Proposition 5.2 Let u

| and let x

(t) = (t, t
0
, x
0
, u

). Then u

is optimal if and only


if c is the inward normal to a hyperplane supporting K(t
f
, t
0
, x
0
) at x

(t
f
) (which implies
that x

(t
f
) is on the boundary of K(t
f
, t
0
, x
0
)).
Exercise 5.6 Prove Proposition 5.2.
Exercise 5.7 (i) Assuming that U is convex, show that | is convex. (ii) Assuming that |
is convex, show that K(t
2
, t
1
, z) is convex.
Remark 5.5 It can be shown that K(t
2
, t
1
, z) is convex even if U is not, provided we enlarge
| to include all bounded measurable functions u : [t
0
, ) U: see Theorem 1A, page 164
of [16].
Geometric interpretation (see Figure 5.3)
Taking t
1
= t
0
in (2.23), we see that, if u

is optimal, i.e., if c = p

(t
f
) is the inward normal to
a hyperplane supporting K(t
f
, t
0
, x
0
) at x

(t
f
) then, for t
0
t t
f
, x

(t) is on the boundary


of K(t, t
0
, x
0
) and p

(t) is the inward normal to a hyperplane supporting K(t, t


0
, x
0
) at x

(t).
This normal is obtained by transporting backwards in time, via the adjoint equation, the
inward normal p

(t
f
) at time t
f
.
Suppose now that the objective function is of the form(x(t
f
)), continuously dierentiable,
instead of c
T
x(t
f
) and suppose K(t
f
, t
0
, x
0
) is convex (see remark above on convexity of
K(t, t
0
, z)).
We want to
minimize (x) s.t. x K(t
f
, t
0
, x
0
)
we know that if x

(t
f
) is optimal, then
(x

(t
f
))
T
h 0 h cl (coTC(x

(t
f
), K(t
f
, t
0
, x
0
)))
Claim: from convexity of K(t
f
, t
0
, x
0
), this implies
(x

(t
f
))
T
(x x

(t
f
)) 0 x K(t
f
, t
0
, x
0
). (5.32)
Copyright c _19932011, Andre L. Tits. All Rights Reserved 125
Calculus of Variations and Optimal Control
Exercise 5.8 Prove the claim.
Note: Again, (x

(t
f
)) is an inward normal to K(t
f
, t
0
, x
0
) at x

(t
f
).
An argument identical to that used in the case of a linear objective function shows that a
version of Pontryagins Principle still holds in this case (but only as a necessary condition,
since (5.32) is merely necessary), with the terminal condition on the adjoint equation being
now
p

(t
f
) = (x

(t
f
)).
Note that the adjoint equation cannot anymore be integrated independently of the state
equation, since x

(t
f
) is needed to integrate p

(backward in time from t


f
to t
0
).
Exercise 5.9 By following the argument in these notes, show that a version of Pontryagins
Principle also holds for a discrete-time linear optimal control problem. (We saw earlier that
it may not hold for a discrete nonlinear optimal control problem. Also see discussion in the
next section of these notes.)
Exercise 5.10 If is convex and U is convex, Pontryagins Principle is again a necessary
and sucient condition of optimality.
More general boundary conditions
minimize c
T
x(t
f
) s.t.
dynamics : x(t) = A(t)x(t) + B(t)u(t) t
0
t t
f
initial condition : G
0
x(t
0
) = b
0
(Let T
0
= x : G
0
x = b
0

nal condition : G
f
x(t
f
) = b
f
(Let T
f
= x : G
f
x = b
f

control constraints : u |
where G
0
and G
f
are xed matrices of appropriate dimensions. We give the following theorem
without proof (see, e.g., [26]).
Theorem 5.3 Let x

(t
0
) T
0
, u

|. Let x

(t) = (t, t
0
, x

(t
0
), u

) and suppose x

(t
f
)
T
f
. Then (i) If (u

, x

(t
0
)) is optimal for (P) then
0
0 and a function p

: [t
0
, t
f
] R
n
,
not both identically zero, satisfying
p

(t) = A
T
(t)p

(t) t
0
t t
f
adjoint equation (5.33)
transversality conditions
_
p

(t
0
) A(G
0
) (tg. plane to T
0
)
p

(t
f
)
0
c A(G
f
) (tg. plane to T
f
)
(5.34)
and the Pontryagin Principle
H(t, x

(t), u

(t), p

(t)) = 1(t, x

(t), p

(t)) (5.35)
holds for all t. (ii) Conversely if there exists
0
> 0 and p

satisfying (5.33), (5.34),


(5.35), then (u

, x

(t
0
)) is optimal for (P). Note that, similar to discrete-time, the second
transversality condition implies that
p

(t
f
) z R
n
:
_
G
f
c
T
_
z = 0
126 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
5.4 Continuous-Time Optimal Control; Nonlinear Systems
5.4 Continuous-Time Optimal Control; Nonlinear Sys-
tems
See [26]
We consider the system
x(t) = f(t, x(t), u(t)) t
0
t t
f
(5.36)
Unlike the linear case, we do not have an explicit expression for (t
f
, t
0
, x
0
, u). We shall
settle for a comparison between the trajectory x

and trajectories x obtained by perturbing


the control u

. By considering strong perturbations of u

we will be able to still obtain a


global Pontryagin Principle, involving a global minimization of the pre-Hamiltonian. Some
proofs will be omitted.
We impose the following regularity conditions on equation (5.36)
(i) for each t [t
0
, t
f
], f(t, , ) : R
n
R
m
R
n
is continuously Frechet dierentiable
in the remaining variables.
(ii) except for a nite subject D [t
0
, t
f
], the functions f,
f
x
,
f
u
are continuous on [t
0
, t
f
]
R
n
R
m
.
(iii) for every nite , , s.t.
[[f(t, x, u)[[ + [[x[[ t [t
0
, t
f
], x R
n
, u R
m
, [[u[[ .
Under these conditions, for any t
1
[t
0
, t
f
], any z R
n
, and any piecewise continuous
u : [t
0
, t
f
] R
m
, (5.36) has a unique solution
x(t) = (t, t
1
, z, u) t
1
t t
f
such that x(t
1
) = z. Let x(t
0
) = x
0
be given. Again, let U R
m
and let
| = u : [t
0
, t
f
] U , u piecewise continuous
be the admissible set.
Note: The constraint on the values of u (see U) is what makes this problem harder than
those tackled by the classical calculus of variations.
Let u

| be optimal and let x

be the corresponding state trajectory. As in the linear case


we must have (necessary condition)
(x

(t
f
))
T
h 0 h cl(coTC(x

(t
f
), K(t
f
, t
0
, x
0
))),
where (x(t
f
)) is the (continuously dierentiable) objective function. However, unlike in the
linear case (convex reachable set), there is no explicit expression for this tangent cone. We
will obtain a characterization for a subset of interest of this cone. This subset will correspond
to a particular type of perturbations of x

(t
f
). The specic type of perturbation to be used
Copyright c _19932011, Andre L. Tits. All Rights Reserved 127
Calculus of Variations and Optimal Control
is motivated by the fact that we are seeking a global Pontryagin Principle, involving a min
over the entire U set.
Let D

be the set of discontinuity points of u

. Let (t
0
, t
f
), , D

D, and let v U.
For > 0, we consider the strongly perturbed control u
,
dened by
u
,
(t) =
_
v for t [ , )
u

(t) elsewhere
A key fact is that, as shown by the following proposition, even though v may be very remote
from u

(), the eect of this perturbed control on x is small. Let x


,
be the trajectory
corresponding to u
,
.
Proposition 5.3
x
,
(t
f
) = x

(t
f
) + h
,v
+ o()
with o satisfying
o()

0 as 0, and with
h
,v
= (t
f
, )[f(, x

(), v) f(, x

(), u

())].
where is the state transition matrix for the system

=
f
x
(t, x

(t), u

(t)).
See, e.g., [16, p. 246-250] for a detailed proof of this result. The gist of the argument is that
(i) to rst order in ,
x
,
() = x

( ) +
_

f(t, x

(t), v)d x

( ) + f(, x

(), v)
and
x

() = x

( ) +
_

f(t, x

(t), u

(t))d x

( ) + f(, x

(), u

(t))
so that
x
,
() x

() [f(, x

(), v) f(, x

(), u

())].
and that (ii) still to rst order in , for t > ,
d
dt
(t) := x
,
(t) x

(t) = f(t, x
,
(t), u

(t)) f(t, x

(t), u

(t))
f
x
(t, x

(t), u

(t))(t).
Exercise 5.11 (t
0
, t
f
), , D

D, t [t
0
, t
f
], v U,
h
,v
(t) TC(x

(t), K(t, t
0
, x
0
)) (5.37)
This leads to the following Pontryagin Principle (necessary condition)
128 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
5.4 Continuous-Time Optimal Control; Nonlinear Systems
Theorem 5.4 Let : R
n
R be continuously dierentiable and consider the problem
minimize (x(t
f
)) s.t.
x(t) = f(t, x(t), u(t)) t
0
t t
f
x(t
0
) = x
0
, u |
Suppose u

| is optimal and let x

(t) = (t, t
0
, x
0
, u

). Let p

(t), t
0
t t
f
, be the
solution of the adjoint equation
p

(t) =
f
x
(t, x

(t), u

(t))
T
p

(t) p

(t
f
) = (x

(t
f
))
=
x
H(t, x

(t), u

(t), p

(t))
with
H(t, x, u, p) = p
T
f(t, x, u) t, x, u, p
Then u

satises the Pontryagin Principle


H(t, x

(t), u

(t), p

(t)) = 1(t, x

(t), p

(t)) (5.38)
for all t [t
0
, t
f
].
Proof (compare with linear case). In view of (5.37), we must have
(x

(t
f
))
T
h
t,v
(t
f
) 0 t [t
0
, t
f
], v U (5.39)
Using the expression we obtained above for h
t,v
, we get
(x

(t
f
))
T
((t
f
, t)[f(t, x

(t), v) f(t, x

(t), u

(t))]) 0 t (t
0
, t
f
), t , D

D, v U.
Introducing the co-state similarly to the linear case, we get
p

(t)
T
f(t, x

(t), u

(t)) p

(t)
T
f(t, x

(t), v) t (t
0
, t
f
), t , D

D, v U.
Remark 5.6 Pontryagins Principle can be used as follows.
(i) Solve the minimization in (5.38) to obtain, for each t, u

(t) as a function of x

(t) and
p

(t) (independently for each t !). (minimization in a nite dimension space).


(ii) Plug it in the state equation. Integrate the direct and adjoint system for x

and p

.
[Note that for the current case, the adjoint equation can be integrated independently.]
Remark 5.7 If u

| is locally optimal, in the sense that x

(t
f
) is a local minimizer for
in K(t
f
, t
0
, x
0
), Pontryagins Principle still holds (with a global minimization over U). Why?
The following exercise generalizes Exercise 2.10.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 129
Calculus of Variations and Optimal Control
Exercise 5.12 Prove that, if f does not explicitly depend on t, then m(t) = 1(t, x

(t),
p

(t)) is constant. Assume that u

is continuously dierentiable on all the intervals over


which it is continuous.
Geometric approach to discrete-time case
We investigate to what extent the approach just used for the continuous-time case can
also be used in the discrete-time case, the payo being the geometric intuition.
A hurdle is immediately encountered: strong perturbations as described above cannot
work in the discrete-time case. The reason is that the time interval during which the control is
varied cannot be made arbitrarily small (it is at least one time step), and thus the smallness
of the perturbation must come from the perturbed value v. Consequently, at every t, u

(t)
can only be compared to nearby values v and a true Pontryagin Principle cannot be obtained.
We investigate what can be obtained by considering appropriate weak variations.
Thus consider the discrete-time system
x
i+1
= x
i
+ f(i, x
i
, u
i
), i = 0, . . . , N 1,
and the problem of minimizing (x
N
), given a xed x
0
and the constraint that u
i
U for
all i. Suppose u

i
, i = 0, . . . , N 1, is optimal, and x

i
, i = 1, . . . , N is the corresponding
optimal state trajectory. Given k 0, . . . , N 1, > 0, and w TC(u

k
, U), consider the
weak variation
(u
k,
)
i
=
_
u

k
+ w + o() i = k
u

i
otherwise
where o() is selected in such a way that u
,i
U for all i, which is achievable due to the
choice of w. The next state value is then given by
(x
k,
)
k+1
= x

k+1
+ f(k, x

k
, u

k
+ w + o()) f(k, x

k
, u

k
) (5.40)
= x

k+1
+
f
u
(k, x

k
, u

k
)w + o(). (5.41)
The nal state is then given by
(x
k,
)
N
= x

N
+ (N, k + 1)
f
u
(k, x

k
, u

k
)w + o().
which shows that h
k,w
:= (N, k +1)
f
u
(k, x

k
, u

k
)w belongs to TC(x

N
, K(N, 0, x
0
)). We now
can proceed as we did in the continuous-time case. Thus
(x

N
)
T
(N, k + 1)
f
u
(k, x

k
, u

k
)w 0 k, w TC(u

k
, U).
Letting p

k
solve the adjoint equation,
p
i
= p
i+1
+
f
x
(i, x
i
, u
i
)p
i+1
with p

N
= (x

N
),
i.e., p

k+1
= (N, k + 1)
T
(x

N
), we get
(p

k+1
)
T
f
u
(k, x

k
, u

k
)w 0 k, w TC(u

k
, U).
130 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
5.4 Continuous-Time Optimal Control; Nonlinear Systems
And dening H(j, , , v) =
T
f(j, , v) we get
H
u
(k, x

k
, p

k+1
, u

k
)w 0 k, w TC(u

k
, U),
which is a mere necessary condition of optimality for the minimization of H(k, x

k
, p

k+1
, u)
with respect to u U. It is a special case of the result we obtained earlier in this chapter
for the discrete-time case.
More general boundary conditions
Suppose now that x(t
0
) is not necessarily xed but, more generally, it is merely con-
strained to satisfy g
0
(x(t
0
)) = 0 where g
0
is a given continuously dierentiable function. Let
T
0
= x : g
0
(x) = 0. (T
0
= R
n
is a special case of this, where the image space of g
0
has
dimension 0. T
0
= x
0
, i.e., g
0
(x) = x x
0
, is the other extreme: xed initial point.) It
should be clear that all the optimality conditions we have derived so far must still hold. We
now obtain an additional condition (which will be of much interest, since we now have one
more degree of freedom). Let x

(t
0
) be the optimal initial point. From now on, the following
additional assumption will be in force:
Assumption.
g
0
x
(x

(t
0
)) is full row rank.
Let
d A
_
g
0
x
(x

(t
0
))
_
= TC(x

(t
0
), T
0
).
Then, for > 0 small enough there exists some o function such that x

(t
0
) +d +o() T
0
.
For > 0 small enough, let x

(t
0
) = x

(t
0
) + d + o(), and consider applying our optimal
control u

(for initial point x

(t
0
)), but starting from x

(t
0
) as initial point. We now invoke
the following result, given as an exercise.
Exercise 5.13 Show that, if d TC(x

(t
0
), T
0
), then
(t
f
, t
0
)d TC(x

(t
f
), K(t
f
, t
0
, T
0
)),
where is as in Proposition 5.3. (Hint: Use the fact that
x(t)

follows linearized dynamics.


See, e.g., Theorem 10.1 in [13].)
It follows that optimality of x

(t
f
) yields
(x

(t
f
))
T
(t
f
, t
0
)d 0 d A
_
g
0
x
(x

(t
0
))
_
,
i.e., since A
_
g
0
x
(x

(t
0
))
_
is a subspace,
((t
f
, t
0
)
T
(x

(t
f
)))
T
d = 0 d A
_
g
0
x
(x

(t
0
))
_
.
Thus
p

(t
0
) A
_
g
0
x
(x

(t
0
))
_
(5.42)
Copyright c _19932011, Andre L. Tits. All Rights Reserved 131
Calculus of Variations and Optimal Control
which is known as a transversality condition. Note that p

(t
0
) is thus no longer free. Indeed,
for each degree of freedom gained on x

(t
0
), we lose one on p

(t
0
).
Now suppose that the initial state is xed but the nal state x

(t
f
), instead of be-
ing entirely free, is possibly constrained, specically, suppose x

(t
f
) is required to satisfy
g
f
(x(t
f
)) = 0, where g
f
is a given continuously dierentiable function. Let T
f
= x :
g
f
(x) = 0. (T
f
= x
f
, with x
f
given, is a special case of this, where g
f
is given by
g
f
(x) x x
f
.) Now should no longer be minimized over the entire reachable set, but
only on its intersection with T
f
. The Pontryagin Principle as previously stated no longer
holds. Rather we can write
(x

(t
f
))
T
h 0 h TC(x

(t
f
), K(t
f
, t
0
, T
0
) T
f
).
We now assume that
TC(x

(t
f
), K(t
f
, t
0
, T
0
) T
f
) = TC(x

(t
f
), K(t
f
, t
0
, T
0
)) TC(x

(t
f
), T
f
). (5.43)
Such assumption amounts to a type of constraint qualication.
Exercise 5.14 Show that, if x
1

2
then TC(x,
1

2
) TC(x,
1
) TC(x,
2
).
Provide an example showing that equality does not always hold.
From now one, the following additional assumption will be in force:
Assumption.
g
f
x
(x

(t
f
)) is full row rank.
We obtain
(x

(t
f
))
T
h 0 h cl(coTC(x

(t
f
), K(t
f
, t
0
, T
0
))) A
_
g
f
x
(x

(t
f
))
_
.
Now let m
f
be the dimension of the image-space of g
f
and dene
R = (1, 0, . . . , 0)
T
R
m
f
+1
and
C =
__
(x

(t
f
))
T
h
g
f
x
(x

(t
f
))h
_
: h cl(coTC(x

(t
f
), K(t
f
, t
0
, T
0
)))
_
Then C is a convex cone and is strictly separated from point R by a non-vertical (i.e., the
rst component of the normal is nonzero) hyperplane (prove it). Thus there exists R
m
f
such that
(x

(t
f
))
T
h +
T
g
f
x
(x

(t
f
))h 1 h cl(coTC(x

(t
f
), K(t
f
, t
0
, T
0
))),
implying, since cl(coTC(x

(t
f
), K(t
f
, t
0
, T
0
))) is a cone,
_
(x

(t
f
)) +
g
f
x
(x

(t
f
))
T

_
T
h 0 h cl(coTC(x

(t
f
), K(t
f
, t
0
, T
0
))).
132 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
5.4 Continuous-Time Optimal Control; Nonlinear Systems
If we impose the condition
p

(t
f
) = (x

(t
f
)) +
g
f
x
(x

(t
f
))
T

we obtain, formally, the same Pontryagin Principle as above. However, since is free, p

(t
f
)
is no longer xed, but rather, it is merely required to satisfy
p

(t
f
) (x

(t
f
)) A
_
g
f
x
(x

(t
f
))
_
.
This is again a transversality condition. It can be shown that, without the constraint qual-
ication (5.43), the same result would hold except that the transversality condition would
become: there exists
0
0, with (p

(t
f
),
0
) not identically zero, such that
p

(t
f
)
0
(x

(t
f
)) A
_
g
f
x
(x

(t
f
))
_
. (5.44)
Finally, it can be shown that both (5.42) and (5.44) hold in the general case when x

(t
0
)
may be non-xed and x

(t
f
) may be non-free.
Integral objective functions
Suppose the objective function, instead of being (x(t
f
)), is of the form
_
t
f
t
0
f
0
(t, x(t), u(t))dt .
This can be handled as a special case of the previous problem. To this end, we consider the
augmented system with state variable x = (x
0
, x) R
1+n
as follows

x(t) =
_
f
0
(t, x(t), u(t))
f(t, x(t), u(t))
_
=

f(t, x(t), u(t)),
x
0
(t
0
) = 0, x
0
(t
f
) free. Now the problem is equivalent to minimizing
( x(t
f
))

= x
0
(t
f
)
with dynamics and constraints of the same form as before. After some simplications (check
the details), we get the following result. We include again the constraints g
0
(x(t
0
)) = 0,
g
f
(x(t
f
)) = 0.
Theorem 5.5 Let (u

, x

0
) be optimal. Suppose
g
0
x
(x

0
) and
g
f
x
(x

f
) are full row rank. Let
u

|, x

0
s.t. g
0
(x

0
) = 0 and let x

(t) = (t, t
0
, x

0
, u

) and suppose that g


f
(x

(t
f
)) = 0.
Then there exists a function p

= (p

0
, p

) : [t
0
, t
f
] R
1+n
, not identically zero, with
(without loss of generality) p

0
(t) = constant 0, satisfying

(t) =
_


f
x
(t, x

(t), u

(t)
_
T
p

(t)
Copyright c _19932011, Andre L. Tits. All Rights Reserved 133
Calculus of Variations and Optimal Control
or, equivalently,
p

(t) =
_
f
x
(t, x

(t), u

(t)
_
T
p

(t) p

0
f
0
x
(t, x

(t), u

(t))
T
p

(t
0
) A(
g
0
x
(x

0
))
p

(t
f
) A(
g
f
x
(x

f
))
_
transversality conditions
Furthermore

H(t, x

(t), u

(t), p

(t)) =

1(t, x

(t), p

(t))
for all but nitely many t, where we have dened

H(t, x, u, p) = p
0
f
0
(t, x, u) + p
T
f(t, x, u)

1(t, x, p) = inf
vU

H(t, x, v, p)
Finally, if f
0
and f do not depend explicitly on t,
m(t) =

1(t, x

(t), p

(t)) = constant.
Exercise 5.15 Prove the theorem. As in Exercise 2.10, assume that u

is piecewise contin-
uously dierentiable.
Free nal time
Suppose the nal time is itself a decision variable (example: minimum-time problem).
Consider the problem
minimize
_
t
f
t
0
f
0
(t, x(t), u(t))dt subject to
x(t) = f(t, x(t), u(t)) t
0
t t
f
(dynamics)
g
0
(x(t
0
)) = 0, g
f
(x(t
f
)) = 0 (initial and nal conditions)
u | (control constraints)
t
f
t
0
(nal time constraint)
We analyze this problem by converting the variable time interval [t
0
, t
f
] into a xed time
interval [0, 1]. Dene t(s) to satisfy
dt(s)
ds
= (s). (5.45)
To fall back into a known formalism, we will consider s as the new time, and t(s) and (s)
as a new state variables. Making into a state variable instead of a control variable permits
us to require that it be constant (which clearly is allowable, and can only simplify things),
by selecting the corresponding state equation to be
d(s)
ds
= 0, (0) =
0
,
134 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
5.4 Continuous-Time Optimal Control; Nonlinear Systems
where
0
is free. The initial and nal condition on t(s) are
t(0) = t
0
t(1) free
Denoting z(s) = x(t(s)), v(s) = u(t(s)) we obtain the state equation
d
ds
z(s) = (s)f(t(s), z(s), v(s)) 0 s 1
g
0
(z(0)) = 0, g
f
(z(1)) = 0.
Now suppose that (u

, t

f
, x

0
) is optimal for the original problem. Then the corresponding
(v

0
, x

0
), with

= t

f
t
0
(constant!) is optimal for the transformed problem. Expressing
the known conditions for this problem and performing some simplication, we obtain the
following result.
Theorem 5.6 Same as before with the additional result

1(t

f
, x

(t

f
), p

(t

f
)) = 0
where t

f
is the optimal nal time. Again, if f
0
and f do not explicitly depend on t, then

1(t, x

(t), p

(t)) = constant = 0 t
Exercise 5.16 Prove the theorem.
Minimum time problem
Consider the following special case of the previous problem (with xed initial and nal
states).
minimize
_
t
f
t
0
(1)dt subject to
x(t) = f(t, x(t), u(t))
x(t
0
) = x
0
, x(t
f
) = x
f
u |, t
f
t
0
(t
f
free
The previous theorem can be simplied to give the following.
Theorem 5.7 Let t

f
t
0
and u

| be optimal. Let x

be the corresponding trajectory.


Then there exists a function p

: [t
0
, t

f
] R
n
, not identically zero such that
p

(t) =
_
f
x
(t, x

(t), u

(t))
_
T
p

(t) t
0
t t

f
[p

(t
0
), p

(t
f
) free]
H(t, x

(t), p

(t), u

(t)) = 1(t, x

(t), p

(t))
1(t

f
, x

(t

f
), p

(t

f
)) 0
Copyright c _19932011, Andre L. Tits. All Rights Reserved 135
Calculus of Variations and Optimal Control
with
H(t, x, p, u) = p
T
f(t, x, u) (no f
0
)
1(t, x, p) = inf
vU
H(t, x, p, u)
Also, if u

(t) is bounded and f does not depend explicitly on t then 1(t, x

(t), p

(t)) is
constant.
Exercise 5.17 Prove the theorem.
Remark 5.8 Note that p is determined only up to a constant scalar factor.
Example 5.2 (see [26]). Consider the motion of a point mass
m x + x = u , x(t), u(t) R , m > 0
Suppose that u(t) is constrained by
[u(t)[ 1 t
Starting from x
0
, x
0
we want to reach x = 0, x = 0 in minimum time. Set x
1
= x, x
2
= x,
=

m
> 0, b =
1
m
> 0. Let U = [1, 1]. The state equation is
_
x
1
(t)
x
2
(t)
_
=
_
0 1
0
_ _
x
1
(t)
x
2
(t)
_
+
_
0
b
_
u(t) (5.46)
Since the system is linear, and the objective function is linear in (x, t
f
), the Pontryagin
Principle is a necessary and sucient condition of optimality. We seek an optimal control
u

with corresponding state trajectory x

.
(i) The pre-Hamiltonian H (no need for augmented

H here) is
H(t, x, p, u) = p
T
(Ax + Bu) = p
T
[x
2
; x
2
+ bu] = (p
1
p
2
)x
2
+ bp
2
u.
Thus Pontragins Principle gives (since b > 0)
u

(t) = +1 if p

2
(t) < 0
1 if p

2
(t) > 0
anything if p

2
(t) = 0
(ii) We seek information on p

(). The adjoint equation is


_
p

1
(t)
p

2
(t)
_
=
_
0 0
1
_ _
p

1
(t)
p

2
(t)
_
(5.47)
Note that we initially have no information on p

(0) or p

(t
f
), since x

(0) and x

(t
f
)
are xed. (5.47) gives
p

1
(t) = p

1
(0)
p

2
(t) =
1

1
(0) + e
t
(
1

1
(0) + p

2
(0))
We now need to determine p

(0) from the knowledge of x

(0) and x

(t
f
), i.e., we have
to determine p

(0) such that the corresponding u

steers x

to the target (0, 0).


136 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
5.4 Continuous-Time Optimal Control; Nonlinear Systems
(iii) For this, observe that, as p

2
is monotonic in t, u

(t) can change its value at most once,


except if p

2
(t) = 0 t. Clearly, the latter cant occur since (check it) it would imply
that p

is identically zero, which the theorem rules out. The following cases can arise:
case 1. p

1
(0) + p

2
(0) > 0 p

2
strictly monotonic increasing
then either
u

(t) = + 1 t
or
u

(t) =
_
+1 t <

t for some

t
1 t >

t
or
u

(t) = 1 t
case 2. p

1
(0) + p

2
(0) < 0 p

2
strictly monotonic decreasing
then either
u

(t) = 1 t
or
u

(t) =
_
1 t <

t for some

t
+1 t >

t
or
u

(t) = + 1 t
case 3. p

1
(0) + p

2
(0) = 0 p

2
constant, p

1
(0) ,= 0 (otherwise p() 0)
p

2
(t) =
1

1
(0) t
then either
u

(t) = 1 t
or
u

(t) = 1 t
Thus we have narrowed down the possible optimal controls to the controls having the fol-
lowing property.
[u

(t)[ = 1 t
u

(t) changes sign at most once.


No additional information can be extracted from the adjoint equation.
(iv) We now decide to forget about the question of determining p

(0). We will investigate


the question of which among the controls just obtained steers the given initial point
to the origin. It turns out that exactly one such control will do the job, hence will be
optimal.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 137
Calculus of Variations and Optimal Control
We proceed as follows. Starting from x = (0, 0), we apply all possible controls backward in
time and check which yield the desired initial condition. Let (t) = x(t

t).
1. u

(t) = 1 t
Reversing time, we obtain the system

1
(t) =
2
(t)

2
(t) =
2
(t) b
with
1
(0) =
2
(0) = 0. This gives

1
(t) =
b

(t +
e
t
1

),
2
(t) =
b

(1 e
t
) < 0 t > 0.
Also, eliminating t yields

1
=
1

_
b

log(1

b

2
)
2
_
.
Thus
1
is increasing,
2
is decreasing (see curve OA in Figure 5.4).
2. u

(t) = 1 t

1
(t) =
b

(t +
e
t
1

),
2
(t) =
b

(1 e
t
)
Also, eliminating t yields

1
=
1

log(1 +

b

2
) +
2
_
.
Thus
1
is decreasing,
2
is increasing initially (see curve OB in Figure 5.4)
Suppose now that u

(t) = +1 until some time



t, 1 afterward. Then the trajectory for
is of the type OCD (
1
must keep increasing while
2
< 0). If u

(t) = 1 rst, then


+1, the trajectory is of the type OEF.
The reader should convince himself/herself that one and only one trajectory passes
through any point in the plane. Thus the given control, inverted in time, must be
the optimal control for initial conditions at the given point (assuming that an optimal
control exists).
We see then that the optimal control u

(t) has the following properties, at each time t


if x

(t) is above BOA or on OB u

(t) = 1
if x

(t) is below BOA or on OA u

(t) = 1
Thus we can synthesize the optimal control in feedback form: u

(t) = (x

(t)) where the


function is given by
(x
1
, x
2
) =
_
1 if (x
1
, x
2
) is below BOA or on OA
1 above BOA or on OB
BOA is called the switching curve.
138 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
5.4 Continuous-Time Optimal Control; Nonlinear Systems
A
1
2
0
-1
E
B
C
D
F
+1
-1
+1
Figure 5.4: ( < 1 is assumed)
Example 5.3 Linear quadratic regulator
x(t) = A(t)x(t) + B(t)u(t) (dynamics) A(), B() piecewise continuous
x(0) = x
0
given
we want to
minimize J =
1
2
_
1
0
_
x(t)
T
L(t)x(t) + u(t)
T
R(t)u(t)
_
dt
where L(t) = L(t)
T
_ 0 t, R(t) = R(t)
T
0 t, say, both continuous, and where the
nal state is free. The pre-Hamiltonian

H is given by

H(x, p, u, t) =
1
2
p
0
(x
T
L(t)x + u
T
R(t)u) + p
T
(A(t)x + B(t)u).
Let
p

(t) =
_
p

0
p

(t)
_
, 0 (5.48)
be as given by the Pontryagin Principle. We know that p

0
is a constant. We rst show that
p

0
,= 0. By contradiction. Suppose p

0
= 0. The adjoint equation is then
p

(t) = A
T
(t)p

(t)
1
2
p

x
_
x

(t)
T
L(t)x

(t) + u

(t)
T
R(t)u

(t)
_
T
= A
T
(t)p

(t)
with the terminal condition p

(1) = 0. Thus p

(t) = 0 t, so that p

(t) =
_
p

0
p

(t)
_
= 0 t
which contradicts (5.48). Thus p

0
,= 0. We can rescale p

(t) and dene p

0
= 1.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 139
Calculus of Variations and Optimal Control
Since U = R
n
, the Pontryagin Principle now yields, as R(t) > 0 t,
R(t)u

(t) + B
T
(t)p

(t) = 0 .
I.e., in this case, we explicitly solve for u

(t) in terms of p

(t).
Thus
u

(t) = R(t)
1
B(t)
T
p

(t).
The adjoint equation is
p

(t) = A
T
(t)p

(t) L(t)x

(t)
and the state eq.
x

(t) = A(t)x

(t) B(t)R(t)
1
B(t)
T
p

(t)
_

_
(S)
with p

(1) = 0, x

(0) = x
0
. Integrating, we can write
_
p

(1)
x

(1)
_
=
_

11
(1, t)
12
(1, t)

21
(1, t)
22
(1, t)
_ _
p

(t)
x

(t)
_
since p

(1) = 0, the rst row yields


p

(t) =
11
(1, t)
1

12
(1, t)x

(t) (5.49)
provided
11
(1, t) is non singular t. This can be proven to be the case (since L(t) is positive
semi-denite; see Theorem 2.2). Let
K(t) =
11
(1, t)
1

12
(1, t)
We now show that K(t) satises a fairly simple equation. Note that K(t) does not depend
on the initial state x
0
. From (5.49), p

(t) = K(t)x

(t). Dierentiating, we obtain


p

(t) =

K(t)x

(t) K(t)[A(t)x

(t) B(t)R(t)
1
B(t)
T
p

(t)]
= (

K(t) K(t)A(t) K(t)B(t)R(t)
1
B(t)
T
K(t))x

(t) (5.50)
On the other hand, the rst equation in (S) gives
p

(t) = A(t)
T
K(t)x

(t) L(t)x

(t) (5.51)
Since K(t) does not depend on the initial state x
0
, this implies

K(t) = K(t)A(t) A(t)


T
K(t) K(t)B(t)R(t)
1
B(t)
T
K(t) + L(t) (5.52)
For the same reason, p(1) = 0 implies K(1) = 0. Equation (5.52) is a Riccati equation. It
has a unique solution, which is a symmetric matrix.
Note that we have
u

(t) = (R(t)
1
B(t)
T
K(t))x

(t)
which is an optimal feedback control law. This was obtained in Chapter 2 using elementary
arguments.
140 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
5.5 Dynamic programming
5.5 Dynamic programming
(see [26, 11])
This approach to optimal control compares the optimal control to all controls, thus yields
sucient conditions. This gives a verication type result. Disadvantage: computationally
very demanding.
1. Discrete time
We consider the problem
minimize
N1

i=0
f
0
(i, x
i
, u
i
) + (x
N
)
s.t. x
i+1
= f(i, x
i
, u
i
) i = 0, . . . , N 1 (P)
x(0) = x
0
u
i
U
i
, i = 0, . . . , N 1,
x
i
X,
where U
i
, X are some arbitrary sets. The key idea is to embed problem (P) into a family of
problems, with all possible initial times k 0, . . . , N 1 and initial conditions x X
minimize
N1

i=k
f
0
(i, x
i
, u
i
) + (x
N
)
s.t.
x
i+1
= f(i, x
i
, u
i
), i = k, k + 1, . . . , N 1, (P
k,x
)
x
k
= x,
u
i
U
i
, i = k, k + 1, . . . , N 1,
x
i
X i = k, . . . , N 1.
Lemma 5.1 If u

k
, . . . , u

N1
is optimal for (P
k,x
) with corresponding trajectory x

k
= x,
x

k+1
, . . . , x

N
and if l k, . . . , N 1, then
u

l
, u

l+1
, . . . , u

N1
is optimal for (P
l,x

l
).
Proof. Suppose not. Then there exists u
l
, . . . , u
N1
, with corresponding trajectory x
l
= x

l
,
x
l+1
, . . ., x
N
, such that
N1

i=l
f
0
(i, x
i
, u
i
) + ( x
N
) <
N1

i=l
f
0
(i, x

i
, u

i
) + (x

N
) (5.53)
Consider then the control u
k
, . . . , u
N1
with
u
i
=
_
u

i
i = k, . . . , l 1
u
i
i = l, . . . , N 1
Copyright c _19932011, Andre L. Tits. All Rights Reserved 141
Calculus of Variations and Optimal Control
and the corresponding trajectory x
k
= x, . . . , x
N
, with
x
i
= x

i
i = k, . . . , l
x
i
i = l + 1, . . . , N
The value of the objective function corresponding to this control is
N1

i=k
f
0
(i, x
i
, u
i
) + ( x
N
)
=
l1

i=k
f
0
(i, x

i
, u

i
) +
N1

i=l
f
0
(i, x
i
, u
i
) + ( x
N
)
. .
<
l1

i=k
f
0
(i, x

i
, u

i
) +
N1

i=l
f
0
(i, x

i
, u

i
) + (x

N
) from (5.54)
=
N1

i=k
f
0
(i, x

i
, u

i
) + (x

N
)
so that u
i
is better than u

i
. Contradiction.
This leads to the main theorem of dynamic programming, Bellmans Principle of Opti-
mality. Assume (P
k,x
) has an optimal control for all k, x. Dene V (k, x) = minimum value
of objective function for (P
k,x
), and let V (N, x) = (x). V is known as the optimal value
function.
Theorem 5.8
V (k, x) = minf
0
(k, x, v) + V (k + 1, f(k, x, v))[v U
k
(5.55)
x X, 0 k N 1
(5.56)
and v U
k
is a minimizer for (5.55) if and only if it is an optimal control at time k.
Proof.
(i) We rst show that the value V (k, x) is achieved by the right-hand side if an optimal
control u = u

k
is used at time k. Indeed, from the denition of V (k, x)
V (k, x) =
N1

i=k
f
0
(i, x

i
, u

i
) + (x

N
)
where u

k
, . . . , u

N1
is an optimal control for (P
k,x
) and where x

k
= x, x

k+1
, . . . , x

N
is
as produced by u

k
, . . . , u

N1
. We can write
142 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
5.5 Dynamic programming
V (k, x) = f
0
(k, x, u

k
) +
N1

i=k+1
f
0
(i, x

i
, u

i
) + (x

N
)
. .
= V (k + 1, x

k+1
)
= V (k + 1, f(k, x, u

k
))
This is because, in view of the lemma above, u

k+1
, . . . , u

N1
is optimal for (P
k+1,x

k+1
).
(ii) We now show that any control u that is not an optimal control at time k, yields a
value larger than V (k, x). For this, let u
k+1
, . . . , u
N1
be optimal for times k + 1 to
N 1 if u has been used at time k, i.e., u
k+1
, . . . , u
N1
is optimal for (P
k+1,f(k,x,u)
)
and let x
k+1
= f(k, x, u), x
k+2
, . . . , x
N
be the corresponding trajectory. Then, again
by denition of V (k, x)
V (k, x) < f
0
(k, x, u) +
N1

i=k+1
f
0
(i, x
i
, u
i
) + ( x
N
)
. .
=V (k+1, x
k+1
)=V (k+1,f(k,x,u))
This proves the result.
Remark 5.9 This result would not hold for more general objective functions, e.g., if some
f
i
s involve several x
j
s or u
j
s.
Corollary 5.1 Suppose that, for k = 0, . . . , N 1, (k, ) : X U
k
is such that
f
0
(k, x, (k, x)) + V (k + 1, f(k, x, (k, x)) = min
uU
k
f
0
(k, x, u) + V (k + 1, f(k, x, u) (5.57)
Then is an optimal feedback control law.
Remark 5.10
1. Equation (5.57), with boundary condition V (N, x) = (x), can be solved recursively,
backwards from time N, for V (k, x) and (k, x). However it should be stressed that
since for a given initial condition x(0) = x
0
the optimal trajectory is not known a priori,
V (k, x) must be computed for all x at every time k (or at least for every possibly
optimal x). Thus this approach is very CPU-demanding.
2. No dierentiability was assumed.
2. Continuous time
Consider the problem
minimize
_
t
f
0
f
0
(t, x(t), u(t))dt + (x(t
f
))
s.t. x(t) = f(t, x(t), u(t)), x(0) = x
0
given, u |
Copyright c _19932011, Andre L. Tits. All Rights Reserved 143
Calculus of Variations and Optimal Control
where U is time-independent and , f
0
, f satisfy same smoothness conditions as before. Let
V (t, x) be the minimum value of the objective function (assumed to exists), starting from
state x at time t, and let V (t
f
, x) = (x). An argument similar to that used in the discrete-
time case yields, for any [0, t
f
t]
V (t, x) = min
_
_
t+
t
f
0
(, x(), u())d + V (t + , x(t + )) [ u : [t, t + ] U, u |
_
(5.58)
where x() is the trajectory generated by u, i.e.
_
_
_

x () = f(, x(), u()) t


x(t) = x
Under dierentiability assumption of V (, ) this can be pushed further (this was not possible
in discrete time). Thus suppose that V (t, x) is dierentiable in both arguments. Then, for
0, (5.58) yields (examine this carefully!)
V (t, x) = min
_
f
0
(t, x, v) + V (t, x) +
V
t
(t, x) +
V
x
(t, x)f(t, x, v) + o() : v U
_
with
o()
[[
0 as 0 (notice the minimization is now over U). Dividing by and letting
0, we get
_
V
t
(t, x) + min
uU
_
f
0
(t, x, u) +
V
x
(t, x)f(t, x, u)
_
= 0
V (t
f
, x) = (x) x (HJB)
This partial dierential equation for V (, ) is called Hamilton-Jacobi-Bellman equation.
If we dene (note that p
0
is now = 1) t, x, p, u

H(t, x, p, u) = f
0
(t, x, u) + p
T
f(t, x, u)
and

1(t, x, p) = inf
uU

H(t, x, p, u)
(HJB) can be written as
_
V
t
(t, x) =

1
_
t, x,
V
x
(t, x)
T
_
V (t
f
, x) = (x)
(Note the formal similarity with the Pontryagin Principle, if we identify p

with
V
x
.) We
now show that (HJB) is a sucient condition of optimality, i.e., that if V (, ) satises (HJB),
it must be the value function. (In discrete time this was obvious since the solution to (5.55)
was clearly unique, given the nal condition V (N, x) = (x).)
Theorem 5.9 Suppose V (, ), dierentiable, satisfying (HJB) together with the boundary
condition. Suppose (, ) piecewise continuous in t and Lipschitz in x satisfying

H
_
t, x,
V
x
(t, x)
T
, (t, x)
_
=

1
_
t, x,
V
x
(t, x)
T
_
x, t (5.59)
Then is an optimal feedback control law and V is the value function.
144 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
5.5 Dynamic programming
Proof. Let

t < t
f
and x R
n
be arbitrary. Let u |, yielding x() when x(

t) = x (initial
condition). Also let x

() satisfy
x

() = f(, x

(), (, x

()))
x

t) = x
(due to assumptions on , there is a unique solution) and let
u

() = (, x

()).
Then we can write
( x(t
f
)) = V (t
f
, x(t
f
))
= V (t
f
, x(t
f
)) V (

t, x(

t)) + V (

t, x)
=
_
t
f

t
dV
dt
(, x())d + V (

t, x)
=
_
t
f

t
_
V
t
(, x()) +
V
x
(, x())f(, x()), u()
_
d + V (

t, x) (5.60)
V (

t, x)
_
t
f

t
f
0
(, x(), u())d from (HJB)
We also have
(x

(t
f
)) = V (t
f
, x

(t
f
)) V (

t, x

t)) + V (

t, x)
=
_
t
f

t
dV
dt
(, x

())d + V (

t, x)
=
_
t
f

t
_
V
t
(, x

()) +
V
x
(, x

())f(, x

(), u

())
_
d + V (

t, x)(5.61)
= V (

t, x)
_
t
f

t
f
0
(, x

(), u

())d
= V (

t, x)
_
t
f

t
f
0
(, x

(), (, x

())d
(5.60) and (5.61) yield
V (

t, x) =
_
t
f

t
f
0
(, x

(), (, x

()))d + (x

(t
f
))
_
t
f

t
f
0
(, x(), u())d + ( x(t
f
))
Thus V (, ) is the value function and is optimal.
Remark 5.11 (HJB) is a partial dierential equation. Thus its solution is very CPU-
demanding.
Exercise 5.18 Solve the linear quadratic regulator problem using HJB (see Example 5.3).
Copyright c _19932011, Andre L. Tits. All Rights Reserved 145
Calculus of Variations and Optimal Control
146 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
Appendix A
Generalities on Vector Spaces
WARNING. This appendix (and to a lesser extent the second appendix) contains material
signicantly beyond what is needed for this course. We hope it will be helpful to many
students in their research and in more advanced courses.
References: [19], [6, Appendix A]
Denition A.1 Let F = R or C and let V be a set. V is a vector space (linear space)
over F if two operations, addition and scalar multiplication, are dened, with the following
properties
(a) x, y V, x + y V and V is an Abelian (aka commutative) group for the addition
operation, i.e., + is associative and commutative, there exists an additive identity
and every x V has an additive inverse x.
(b) F, x V, x V and
(i) x V, , F
1x = x, (x) = ()x, 0x = , =
(ii) x, y V, , F
(x + y) = x + y
( + )x = x + x
If F = R, V is said to be a real vector space. If F = C, it is said to be a complex vector
space. Elements of V are often referred to as vectors or as points.
Exercise A.1 Let x V , a vector space. Prove that x + x = 2x.
In the context of optimization and optimal control, the primary emphasis is on real vector
spaces (i.e., F = R). In the sequel, unless explicitly indicated otherwise, we will assume this
is the case.
Example A.1 R, R
n
, R
nm
(set of nn real matrices); the set of all univariate polynomials
of degree less than n; the set of all continuous functions f : R
n
R
k
; the set C[a, b] of
all continuous functions over an interval [a, b] R. (All of these with the usual + and
operations.) The 2D plane (or 3D space), with an origin (there is no need for coordinate
axes!), with the usual vector addition (parallelogram rule) and multiplication by a scalar.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 147
Generalities on Vector Spaces
Exercise A.2 Show that the set of functions f : R R such that f(0) = 1 is not a vector
space.
In the sequel, we will often write 0 for .
Denition A.2 A set S V is said to be a subspace of V if it is a vector space in its own
right with the same + and operations as in V .
Denition A.3 Let V be a linear space. The family of vectors x
1
, . . . , x
n
V is said to
be linearly independent if any relation of the form

1
x
1
+
2
x
2
+ . . . +
n
x
n
= 0
implies

1
=
2
= . . . =
n
= 0.
Given a nite collection of vectors, its span is given by
sp (b
1
, . . . , b
n
) :=
_
n

i=1

i
b
i
:
i
R, i = 1, . . . , n
_
.
Denition A.4 Let V be a linear space. The family of vectors b
1
, . . . , b
n
V is
said to be a basis for V if (i) b
1
, . . . , b
n
is a linearly independent family, and (ii)
V = sp(b
1
, . . . , b
n
).
Denition A.5 For i = 1, . . . , n, let e
i
R
n
be the n-tupe consisting of all zeros, except
for a one in position i. Then e
1
, . . . , e
n
is the canonical basis for R
n
.
Exercise A.3 The canonical basis for R
n
is a basis for R
n
.
Exercise A.4 Let V be a vector space and suppose b
1
, . . . , b
n
is a basis for V . Prove
that, given any x V , there exists a unique n-tuple of scalars,
1
, . . . ,
n
such that
x =

n
i=1

i
b
i
. (Such n-tuple referred to as the coordinate vector of x in basis b
1
, . . . , b
n
.)
Exercise A.5 Suppose b
1
, . . . , b
n
and b

1
, . . . , b

m
both form bases for V . Then m = n.
Denition A.6 If a linear space V has a basis consisting of n elements then V is said to
be nite-dimensional or of dimension n. Otherwise, it is said to be innite-dimensional.
Example A.2 Univariate polynomials of degree < n form an n-dimensional vector space.
Points in the plane, once one such point is dened to be the origin (but no coordinate system
has been selected yet), form a 2-dimensional vector space. C[a, b] is innite-dimensional.
Exercise A.6 Prove that the vector space of all univariate polynomials is innite-dimensional.
148 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
Generalities on Vector Spaces
Suppose V is nite-dimensional and let b
1
, . . . , b
n
V be a basis. Then, given any x V
there exists a unique n-tuple (
1
, . . . ,
n
) R
n
(the coordinates, or components of x) such
that x =
n

i=1

i
b
i
(prove it). Conversely, to every (
1
, . . . ,
n
) R
n
corresponds a unique
x =

n
i=1

i
b
i
V . Moreover the coordinates of a sum are the sums of the corresponding
coordinates, and similarly for scalar multiples. Thus, once a basis has been selected, any
n-dimensional vector space (over R) can be thought of as R
n
itself. (Every n-dimensional
vectors space is said to be isomorphic to R
n
.)
Denition A.7 Let V be a vector space. The function , : V V F is called an
inner product (scalar product) if
x, y + z = x, y +x, z x, y, z, V
x, y = x, y F, x, y V
x, x > 0 x ,=
y, x = x, y x, y V
Remark A.1 Some authors use a sightly dierent denition for the inner product, with the
second condition replaced by x, y = x, y, or equivalently (given the fourth condition)
x, y = x, y. Note that the dierence is merely notational, since (x, y) : y, x satises
such denition. The denition given here has the advantage that it is satised by the
standard inner product in R
n
x
T
y = x

y, rather than by the slightly less friendly x


T
y.
Example A.3 Let V be nite-dimensional, and let b
i

n
i=1
be a basis. Then x, y =
n

i=1

i
,
where R
n
and R
n
are the vectors of coordinates of x and y in basis b
i

n
i=1
, is an
inner product. It is known as the Euclidean inner product associated to basis b
i

n
i=1
.
The following exercise characterizes all inner products over R
n
.
Exercise A.7 Prove the following statement. A mapping , : R
n
R
n
R is an inner
product over R
n
if and only if there exists a symmetric positive denite matrix M such that
x, y = x
T
My x, y R
n
.
(Here, following standard practice, the same symbol x is used for a point in R
n
and for the
column vector of its components, and the T superscript denotes transposition of that vector.)
Note that, if M = M
T
0, then M = A
T
A for some square nonsingular matrix A, so that
x
T
Mu = (Ax)
T
(Ay) = x
T
y

, where x

and y

are the coordinates of x and y in a new basis.


Hence, up to a change of basis, an inner product over R
n
always takes the form x
T
y.
Exercise A.8 Consider a plane (e.g., a blackboard or sheet of paper) together with a point in
that plane declared to be the origin. With an origin in hand, we can add vectors (points) in the
plane using the parallelogram rule, and multiply vectors by scalars, and it is readily checked
that all vector space axioms are satised; hence we have a vector space V . Two non-collinear
vectors e
1
and e
2
of V form a basis for V . Any vector x V is now uniquely specied by its
Copyright c _19932011, Andre L. Tits. All Rights Reserved 149
Generalities on Vector Spaces
components in this basis; let us denote by x
E
the column vector of its components. Now, let
us say that two vectors x, y V are perpendicular if the angle (x, y) between them (e.g.,
measured with a protractor on your sheet of paper) is /2, i.e., if cos (x, y) = 0. Clearly,
in general, (x
E
)
T
y
E
= 0 is not equivalent to x and y being perpendicular. (In particular,
of course, (e
E
1
)
T
e
E
2
= 0 (since e
E
1
= [1, 0]
T
and e
E
2
= [0, 1]
T
), while e
1
and e
2
may not be
perpendicular to each other.) Question: Determine a positive denite matrix S such that
x, y
S
:= (x
E
)
T
Sy
E
= 0 if and only if x and y are perpendicular. Discuss whether (and
how) you think this may generalize to 3D space, and to higher dimension spaces; an educated
guess is good enough, as computations may be messy.
Example A.4 V = C[t
0
, t
1
], the space of all continuous functions from [t
0
, t
1
] to R, with
x, y =
t
1
_
t
0
x(t)y(t) dt.
This inner product is known as the L
2
inner product.
Example A.5 V = C[t
0
, t
1
]
m
, the space of continuous functions from [t
0
, t
1
] to R
m
. For
x() = (
1
(), . . . ,
m
()), y = (
1
(), . . . ,
m
())
x, y =
t
1
_
t
0
m

i=1

i
(t)
i
(t) dt =
t
1
_
t
0
x(t)
T
y(t) dt.
This inner product is again known as the L
2
inner product. The same inner product is valid
for the space of piecewise-continuous functions | considered in Chapter 2.
Fact. (Cauchy-Bunyakovskii-Schwarz inequality) If V is a vector space with inner product
, ,
[x, y[
2
x, x y, y x, y V.
Moreover both sides are equal if and only if x = or y = x for some R.
Exercise A.9 Prove the fact. Hint: x + y, x + y 0 R.
Gram-Schmidt ortho-normalization
Let V be a nite-dimensional inner product space and let b
1
, . . . , b
n
be a basis for V . Let
u
1
= b
1
, e
1
=
u
1
|u
1
|
2
u
k
= b
k

k1

i=1
b
k
, e
i
e
i
, e
k
=
u
k
|u
k
|
2
, k = 2; . . . , n.
Then e
1
, . . . , e
n
is an orthonormal basis for V , i.e., |e
i
|
2
= 1 for all i, and e
i
, e
j
= 0 for
all i ,= j (Check it).
150 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
Generalities on Vector Spaces
Denition A.8 Given x, y V, x and y are said to be orthogonal if x, y = 0. Given a
set S V , the set
S

= x V : x, s = 0 s S .
is called orthogonal complement of S. It is a subspace.
Exercise A.10 S (S

.
Denition A.9 A norm on a vector space V is a function | |: V R with the following
properties
(i) |x| > 0 x V, x ,=
(ii) R, x V, |x| = [[|x|
(iii) x, y V, |x + y| |x| +|y|
In particular, if , is an inner product on V , then the function | | given by
|x| = x, x
1/2
is a norm on V (the norm derived from the inner product). (Check it.) A normed vector
space is a pair (V, | |) where V is a vector space and | | is a norm on V . Often, when
the specic norm is irrelevant or clear from the context, we simply refer to normed vector
space V .
Example A.6 In R
n
, [[x[[
1
=
n

i=1
[x
i
[, [[x[[
2
= (
n

i=1
(x
i
)
2
)
1/2
, [[x[[
p
= (
n

i=1
(x
i
)
p
)
1/p
, p
[1, ), [[x[[

= max
i
[x
i
[; in the space of bounded continuous functions from R R,
[[f[[

= sup
t
[f(t)[; in C[0, 1], |f|
1
=
1
_
0
[f(t)[dt.
Note that the p-norm requires that p 1. Indeed, when p < 1, the triangle inequality
does not hold. E.g., take p = 1/2, x = (1, 0), y = (0, 1).
For a given vector space, it is generally possible to dene many dierent norms.
Denition A.10 Two norms | |
a
and | |
b
on a same vector space V are equivalent if
there exist N, M > 0 such that x V, N[[x[[
a
[[x[[
b
M[[x[[
a
.
Exercise A.11 Check that the above is a bona de equivalence relationship, i.e., that it is
reexive, symmetric and transitive.
Once the concept of norm has been introduced, one can talk about balls and converging
sequences.
Denition A.11 Given a normed vector space (V, | |), a sequence x
n
V is said to be
convergent (equivalently, to converge) to x

V if |x
n
x

| 0 as n .
Copyright c _19932011, Andre L. Tits. All Rights Reserved 151
Generalities on Vector Spaces
Remark A.2 Examples of sequences that converge in one norm and not in another are well
known. For example, it is readily checked that, in the space P of univariate polynomials (or
equivalently of scalar sequences with only nitely many nonzero terms), the sequence z
k

(i.e., the kth term in the sequence is the monomial given by the kth power of the variable)
does not converge in the sup norm (sup of absolute values of coecients) but converges to
zero in the norm [[p[[ =

1
i
p
i
where p
i
is the coecient of the ith power term. Examples
where a sequence converges to two dierent limits in two dierent norms are more of a
curiosity. The following one is due to Tzvetan Ivanov from Catholic University of Louvain
(UCL) and Dmitry Yarotskiy from Ludwig Maximilian Universitaet Muenchen. Consider
the space P dened above, and for a polynomial p P, write p(z) =

i
p
i
z
i
. Consider the
following two norms on P:
|p|
a
= max[p
0
[, max
i1

[p
i
[
i
,
|p|
b
= max[p
0
+

i1
p
i
[, max
i1

[p
i
p
0
[
i
.
(Note that the coecients p
i
of p in the basis 1, z
1
, z
2
, ..., used in norm a, are replaced in
norm b by the coecients of p in the basis 1, z
1
1, z
2
1, ....) As above, consider the
sequence x
k
= z
k
of monomials of increasing power. It is readily checked that x
k
tends
to zero in norm a, but that it tends to the constant polynomial z
0
= 1 in norm b, since
|x
k
1|
b
=
2
k
tends to zero.
Closed sets, open sets
Denition A.12 Let V be a normed vector space. A subset S V is closed (in V ) if
every x V , for which there is a sequence x
n
S that converges to x, belongs to S. A
subset S V is open if its complement is closed. The closure clS of a set S is the smallest
closed set that contains S, i.e., the intersection of all closed sets that contain S (see the next
exercise). The interior intS of a set S is the largest open set that is contained in S, i.e., the
union of all open sets that contain S.
Exercise A.12 Prove the following. The intersection

of an arbitrary (possibly un-


countable) family of closed sets is closed. The union

of an arbitrary (possibly uncount-


able) family of open sets is open.
Exercise A.13 Let V be a normed vector space and let S V . Show that the closure of S
is the set of all limit points of sequences of S that converge in V .
Exercise A.14 Show that a subset S of a normed vector space is open if and only if given
any x S there exists > 0 such that x : |x x| < S.
Exercise A.15 Suppose that | |
a
and | |
b
are two equivalent norms on a vector space
V and let the sequence x
k
V be such that the sequence |x
k
|
1/k
a
converges. Then
the sequence |x
k
|
1/k
b
also converges and both limits are equal. Moreover, if V is a space
of matrices and x
k
is the kth power of a given matrix A, then the limit exists and is the
spectral radius (A), i.e., the radius of the smallest disk centered at the origin containing all
eigenvalues of A.
152 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
Generalities on Vector Spaces
Exercise A.16 In a normed linear space, every nite-dimensional subspace is closed. In
particular, all subspaces of R
n
are closed.
Exercise A.17 Show that, if S is a nite-dimensional subspace of an inner product space,
then S

= S. [Hint: choose an orthogonal basis for S. ]


(Note: If S is a subspace of an arbitrary inner product space V , then S

= clS, the closure


of S (in the topology generated by the norm derived from the inner product).)
Example A.7 Given a positive integer n, the set of polynomials of degree n in one
variable over [0,1] is a nite-dimensional subspace of C[0, 1]. The set of all polynomials in
one variable over [0,1] is an innite-dimensional subspace of C[0, 1]. It is not closed in either
of the norms of Example A.6 (prove it).
Exercise A.18 Given any set S, S

is a closed subspace (in the norm derived from the


inner product).
Denition A.13 A set S in a normed vector space is bounded if there exists > 0 s.t.
S x : |x|
Denition A.14 A subset S of a normed vector space is said to be (sequentially) compact
if, given any sequence x
k

k=0
S, there exists a sub-sequence that converges to a point of
S, i.e., there exists an innite index set K 0, 1, 2, . . . and x

S such that x
k
x

as k , k K.
Remark A.3 The concept of compact set is also used in more general topological spaces
than normed vector spaces, but with a dierent denition (Every open cover includes a
nite sub-cover). In such general context, the concept introduced in Denition A.14 is
referred to as sequential compactness and is weaker than compactness. In the case of
normed vector spaces (or, indeed, of general metric spaces), compactness and sequential
compactness are equivalent.
Fact. (Bolzano-Weierstrass, Heine-Borel). Let S R
n
. Then S is compact if and only if it
is closed and bounded. (For a proof see, e.g., wikipedia.)
Example A.8 The simplest innite-dimensional vector space may be the space P of
univariate polynomials (of all degrees), or equivalently the space space of sequences with
nitely many nonzero entries. Consider P together with the max norm (maximum absolute
value among the (nitely many) non-zero entries). The closed unit ball in P is not compact.
For example, the sequence x
k
, where x
k
is the monomial z
k
, z being the unknown, which
clearly belongs to the unit ball, does not have a converging subsequence. Similarly, the close
unit ball B in
1
(absolutely summable real sequences) is not compact. For example, the
following continuous function is unbounded over B (example due to Nuno Martins):
f(x) = maxx
n
if x
n

1
2
n, and
1
2
+ maxn(x
n

1
2
) otherwise.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 153
Generalities on Vector Spaces
In fact, it is a important result due to Riesz that the closed unit ball of a normed vector
space is compact if and only if the space if nite-dimensional. (See, e.g., [15, Theorem 6,
Ch. 5].)
Denition A.15 Supremum, inmum. Given a set S R, the supremum supS (resp.
inmum infS) of S is the smallest (resp. largest) s such that s s (resp. s s) for all
s S. If there is no such s, then supS = (resp. infS = ). (Every subset of R has a
supremum and an inmum.)
Denition A.16 Let (V, | |
V
) and (W, | |
W
) be normed spaces, and let f : V W. Then
f is continuous at x v if for every > 0, there exists > 0 such that |f(x) f( x)|
W
<
for all x such that |x x|
V
< . If f is continuous at x for all x V , it is said to be
continuous.
Exercise A.19 Prove that, in any normed vector space, the norm is continuous with respect
to itself.
Exercise A.20 Let V be a normed space and let S V be compact. Let f : V R be
continuous. Then there exists x, x S such that
f(x) f(x) f( x) x S
i.e., the supremum and inmum of f(x) : x S are attained.
Exercise A.21 Let f : V R be continuous. Then, for all in R, the sub-level set
x : f(x) is closed.
Denition A.17 Let f : R
n
R
m
and let S R
n
. Then f is uniformly continuous
over S if for all > 0 there exists > 0 such that
x, y S
|x y| <
_
=|f(x) f(y)| <
Exercise A.22 let S R
n
be compact and let f : R
n
R
m
be continuous over S.
Then f is uniformly continuous over S.
We will see that equivalent norms can often be used interchangeably. The following result
is thus of great importance.
Exercise A.23 Prove that, if V is a nite-dimensional vector space, all norms on V are
equivalent. Hint. First select an arbitrary basis b
i

n
i=1
for V . Then show that ||
V
: V R,
dened by |x|
x
:= max [x
i
[, where the x
i
s are the coordinates of x in basis b
i
, is a norm.
Next, show that, if | | is an arbitrary norm, it is a continuous function from (V, | |
V
) to
(R, [ [). Finally, conclude by using the Fact stated just before Denition A.17.
154 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
Generalities on Vector Spaces
Denition A.18 Given a normed vector space (V, | |), a sequence x
n
V is said to be
a Cauchy sequence if |x
n
x
m
| 0 as n, m , i.e., if for every > 0 there exists N
such that n, m N implies |x
n
x
m
| < .
Exercise A.24 Every convergent sequence is a Cauchy sequence.
Exercise A.25 Let x
i
V and |.|
a
and |.|
b
be 2 equivalent norms on V . Then
(i) x
i
converges to x

with respect to norm a if and only if it converges to x

with respect
to norm b.
(ii) x
i
is Cauchy w.r.t. norm a if and only if it is Cauchy w.r.t. norm b
Hence, in R
n
, we can talk about converging sequences and Cauchy sequences without spec-
ifying the norm.
Exercise A.26 Suppose x
k
R
n
is such that, for some b R
n
, |x
k+1
x
k
| b
T
(x
k+1

x
k
) for all k. (I.e., suppose that nonzero b ,= 0 is such that the angle between b and x
k+1
x
k
is bounded away from /2.) Prove that, if b
T
x
k
is bounded, then x
k
converges. Hint:
Show that the sum

N
k=0
|x
k+1
x
k
| remains bounded as N , and that this implies that
x
k
is Cauchy.
Denition A.19 A normed linear space V is said to be complete if every Cauchy sequence
in V converges to a point of V .
R
n
is complete, and hence every nite-dimensional vector space is complete. Complete
normed spaces are called Banach spaces. Complete inner product spaces (with norm derived
from the inner product) are called Hilbert spaces. R
n
is a Hilbert space.
Example A.9 C[0, 1] with the sup norm is Banach. The vector space of polynomials over
[0,1], with the sup norm, is not Banach, nor is C[0, 1] with an L
p
norm, p nite. The space
of square-summable real sequences, with norm derived from the inner product
x, y =

i=1
x
i
y
i
is a Hilbert space.
Exercise A.27 Exhibit an example showing that the inner product space of Example A.4 is
not a Hilbert space.
While the concepts of completeness and closedness are somewhat similar in spirit, they are
clearly distinct. In particular, completeness applies to vector spaces and closedness applies
to subsets of vectors spaces. Yet, for example, the vector space P of univariate polynomials
is not complete under the sup norm. However it is closed (as a subset of itself), and its
closure (in itself) is itself. Indeed, all vector spaces are closed (as subsets of themselves).
Note however that P can also be thought of as a subspace of the (complete under the sup
norm) space C(R) of continuous functions over [0, 1]. Under the sup norm, P is not a closed
(in C(R)) subspace. (Prove it.) More generally, closedness and completeness are related by
the following result.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 155
Generalities on Vector Spaces
Theorem A.1 Let V be a normed vector space, and let W be a Banach space that contains
a subspace V

with the properly that V and V

are isometric normed vector spaces. (Two


normed vector spaces are isometric if they are isomorphic as vector spaces and the isomor-
phism leaves the norm invariant.) Then V

is closed (in W) if and only if V is complete.


Furthermore, given any such V there exists such W and V

such that the closure of V



(in
W) is W itself. (Such W is known as the completion of V . It is isomorphic to a certain
normed space of equivalence classes of Cauchy sequences in V .) In particular, a subspace S
of a Banach space V is closed if and only if it is complete as a vector space (i.e., if and only
if it is Banach space).
Linear maps
You may think of incomplete vector spaces as porous, and of non-closed subspaces as
porous as well, with pores elements of the mother space whenever that space is complete.
Denition A.20 Let V, W be vector spaces. A map L : V W is said to be linear if
L(x
1
+ x
2
) = L(x
1
) + L(x
2
) , R
x
1
x
2
V
Exercise A.28 Let A : V W be linear, where V and W have dimension n and m,
respectively, and let b
V
i
and b
W
i
be bases for V and W. Show that A can be represented
by a matrix, i.e., there exists a matrix M
A
such that, for any x V, y W such that
y = A(x), it holds that v
y
= M
A
v
x
, where v
x
and v
y
are n 1 and m1 matrices (column
vectors) with entries given by the components of x and y, and is the usual matrix product.
The entries in the ith column of M
A
are the components in b
W
i
of Ab
V
i
.
Linear maps from V to W themselves form a vector space /(V, W). Given a linear map
L /(V, W), its range is given by
1(L) = Lx : x V W
and its nullspace (or kernel) by
A(L) = x V : Lx =
W
V.
Exercise A.29 1(L) and A(L) are subspaces of W and V respectively.
Exercise A.30 Let V and W be vector spaces, with V nite-dimensional, and let L : V
W be linear. Then 1(L) is also nite-dimensional, of dimension no larger than that of V .
Denition A.21 A linear map L : V W is said to be surjective if 1(L) = W; it is said
to be injective if A(L) = 0.
Exercise A.31 Prove that a linear map A : R
n
R
m
is surjective if and only if the
matrix that represents it has full row rank, and that it is injective if and only if the matrix
that represents is has full column rank.
156 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
Generalities on Vector Spaces
Denition A.22 Let V, W be normed vector spaces. A linear map L /(V, W) is said to
be bounded if there exists c > 0 such that |Lx|
W
c|x|
V
for all x V . If L is bounded,
the operator norm (induced norm) of L is dened by
|L| = infc : |Lx|
W
c|x|
V
x V
Exercise A.32 Let V be a nite-dimensional normed vector space, and let L be a linear
map over V . Prove that L is bounded.
Exercise A.33 Let V, W be normed vector spaces and let L /(V, W). The following are
equivalent:
(i) L is bounded
(ii) L is continuous over V
(iii) L is continuous at
V
.
Moreover, if L is bounded then A(L) is closed.
Exercise A.34 If V is nite-dimensional, every linear map L /(V, W) is bounded (and,
for any such L, 1(L) is closed since it is nite-dimensional).
Example A.10 Let V be the vector space of continuously dierentiable functions on [0, 1]
with |x| = max
t[0,1]
[x(t)[. Let W = R and let L be dened to Lx = x

(0). Then L is an
unbounded linear map. (Think of the sequence x
k
(t) = sin(kt).) It can be veried that
A(L) is not closed. For example, let x
k
(t) =
kt
3
1+kt
2
. Then x
k
A(L) for all k and x
k
x
with x(t) = t, but L x = 1.
Example A.11 Let V be C[0, 1] with the L
2
norm, let W = R, and let Lx = x(0). Then,
again, L is an unbounded linear map.
Exercise A.35 Show that
|L| = sup
|x|
V
1
|Lx|
W
= sup
x,=
|Lx|
W
|x|
V
= sup
|x|
V
=1
|Lx|
W
.
The set of bounded linear maps from V to W is a vector space. It is denoted by B(V, W).
Exercise A.36 Show that | | as above is a norm on B(V, W)
Exercise A.37 Prove that
|AB| |A| |B| A B(W, Z), B B(V, W)
with AB dened by
AB(x) = A(B(x)) x V.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 157
Generalities on Vector Spaces
Theorem A.2 (Riesz-Frechet Theorem) (e.g., [23, Theorem 4-12]). Let H be a Hilbert
space and let L : H R be a bounded linear map (i.e. a bounded linear functional on H).
Then there exists H such that
L(x) = , x x H.
Direct sums and orthogonal projections
Denition A.23 Let S and T be two subspaces of a linear space V . The sum S + T :=
s + t : s S, t T of S and T is called a direct sum if S T = . The direct sum is
denoted S T.
Exercise A.38 Given two subspaces S and T of V , V = S T if and only if for every
v V there is a unique decomposition v = s + t such that s S and t T.
It can be shown that, if S is a closed subspace of a Hilbert space H, then
H = S S

.
Equivalently, x H there is a unique y S such that x y S

. The (linear) map


P : x y is called orthogonal projection of x onto the subspace S.
Exercise A.39 Prove that if P is the orthogonal projection onto S, then
|x Px| = inf|x s|, s S,
where | | is the norm derived from inner product.
Adjoint of a linear map
Denition A.24 Let V, W be two vector spaces endowed with inner products ,
V
and
,
W
respectively and let L : V W be linear. Let L

be a map: W V having the


property that
L

y, x
V
= y, Lx
W
x V, y W
Then L

is said to be adjoint to L.
Exercise A.40 Whenever L

exists, it is unique and linear.


It can be shown that, if V and W are complete (i.e. Hilbert spaces) and L is bounded, there
always exists an adjoint map L

, and L

is bounded.
Exercise A.41 (A+ B)

= A

+ B

; (A)

= A

; (AB)

= B

; A

= A.
When L

= L (hence V = W), L is said to be self-adjoint.


Exercise A.42 Let L be a linear map. Show that x, Lx =
1
2
x, (L + L

)x for all x.
158 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
Generalities on Vector Spaces
Exercise A.43 Let A be a linear map from V to W, where V and W are nite-dimensional
vector spaces, and let M
A
be its matrix representation in certain basis b
i
V
and b
i
W
. Let
S
n
and S
m
be n n and m m symmetric positive denite matrices. Obtain the matrix
representation of A

under the inner products x


1
, x
2
:=
T
1
S
n

1
and y
1
, y
2
:=
T
1
S
n

1
,
where
k
and
k
, k = 1, 2, are corresponding vectors of coordinates in bases b
i
V
and b
i
W
.
In particular show that if the Euclidean inner product is used for both spaces (i.e., S
n
and
S
m
are both the identity), then M
A
= M
T
A
, so that A is self-adjoint if and only if M
A
is
symmetric.
Exercise A.44 Let | be given by Example A.5. Consider the map L : | R
n
given by
L(u) =
t
1
_
t
0
G()u()d,
with G : [t
0
, t
1
] R
nm
continuous. L is linear. Verify that L has an adjoint L

: R
n
|
given by
L

(x)(t) = G(t)
T
x
Exercise A.45 Consider the linear time-invariant state-space model (A, B, and C are real)
x = Ax + Bu (A.1)
y = Cx (A.2)
and the associated transfer-function matrix G(s) = C(sI A)
1
B (for s C). The time-
invariant state-space model, with input y and output v,
p = A
T
p C
T
y (A.3)
v = B
T
p. (A.4)
is said to be adjoint to (A.1)-(A.2). (Note the connection with system (2.14), when L =
C
T
C.) Show that the transfer-function matrix G

associated with the adjoint system (A.3)-


(A.4) is given by G

(s) = G(s)
T
; in particular, for every R, G

(j) = G(j)

, where
j :=

1 and a superscript denotes the complex conjugate transpose of a matrix (which


is the adjoint with respect to the complex Euclidean inner product u, v =

, where and
are the vectors of coordinates of u and v, and

is the complex conjugate transpose of ).


This justies referring to p as the adjoint variable, or co-state. The triple (A
T
, C
T
, B
T
)
is also said to be dual to (A, B, C).
Exercise A.46 Let ( be a linear map from C[0, 1]
m
to C[0, 1]
p
, m and p positive integers,
dened by
((u)(t) =
_
t
0
G(t, )u()d, t [0, 1],
where the matrix G(t, ) depends continuously on t and . Let C[0, 1]
m
and C[0, 1]
p
be
endowed with the L
2
inner product, i.e., r, s =
_
1
0
r(t)
T
s(t)dt. Prove that ( has an adjoint
(

given by
((

y)(t) =
_
1
t
G(, t)
T
y()d, t [0, 1]. (A.5)
Copyright c _19932011, Andre L. Tits. All Rights Reserved 159
Generalities on Vector Spaces
Further, verify that, with
G(t, ) = C(t)(t, )B()1(t ),
where 1(t) is the unit step and (t, ) is the state-transition matrix associated with a certain
matrix A(t), ( is the mapping from u to y generated by
x(t) = A(t)x(t) + B(t)u(t), x(0) = 0, (A.6)
y(t) = C(t)x(t) (A.7)
and (

is the mapping from y to v generated by


p(t) = A(t)
T
p(t) C(t)
T
y(t), p(1) = 0, (A.8)
v(t) = B(t)
T
p(t). (A.9)
Observe from (1), and from the terminal condition on p, that (

is anticausal, i.e., the


value of the output at any given time in [0, 1] depends only on present and future values
of the input. Also note that, when A, B, and C are constant, the transfer function matrix
associated with ( is G(s) = C(sI A)
1
B and that associated with (

is G

(s), discussed
in Exercise A.45.
Theorem A.3 Let V, W be inner product spaces, let L : V W be linear, and suppose L
does have an adjoint, L

. Then A(L) and A(L

) are closed and


(a) A(L

) = 1(L)

;
(b) A(L

) = A(LL

);
(c) cl(1(L)) = cl(1(LL

)), and if 1(LL

) is closed, then 1(L) = 1(LL

).
Proof. Closedness of A(L) and A(L

) follows from (a).


(a)
y A(L

) L

y =
V
L

y, x = 0 x V
y, Lx = 0 x V
y 1(L)

.
(We have used the fact that, if L

y, x = 0 x V , then, in particular, L

y, L

y =
0, so that L

y =
V
.)
(b) (i)
y A(L

) L

y =
V
LL

y =
W
y A(LL

)
160 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
Generalities on Vector Spaces
(ii)
y A(LL

) LL

y =
V
y, LL

y = 0 L

y, L

y = 0
L

y = 0 y A(L

)
(c) 1(L)

= A(L

) = A(LL

) = 1(LL

. Thus 1(L)

= 1(LL

. The result
follows. Finally, if 1(LL

) is closed, then
1(LL

) 1(L) cl(1(L)) = cl(1(LL

)) = 1(LL

).
clearly implies that 1(LL

) = 1(L).
Let L : V W be a linear map. If W is a Hilbert space and 1(L) is closed (e.g., W = R
n
or V = R
n
), (a) implies that
W = 1(L) A(L

).
Similarly, if V is a Hilbert space and 1(L

) is closed, since L

= L,
V = 1(L

) A(L).
(L)

w
N(L )
*
(L )
*
N(L)

v
L
L
*
Figure A.1: Structure of a linear map
Theorem A.4 Let V, W be inner-product spaces and let L : V W be linear, with adjoint
L

. Then L[
1(L

)
is one-to-one. It is onto if and only if 1(L) = 1(LL

).
Proof. We prove suciency in the second claim; the remainder is left as an exercise. Thus
let w 1(L). Since 1(L) = 1(LL

), it follows that w = LL

w for some w W. But this


implies that w = Lv, with v = L

w. Since L

w 1(LL

), this completes the proof that


L[
1(L

)
is onto.
This leads to the solution of the linear least squares problem, stated next.
Theorem A.5 Let V, W be inner-product spaces and let L : V W be linear, with adjoint
L

. Suppose that 1(L) = 1(LL

). Let w 1(L). Then the problem


minimize v, v s.t. Lv = w (A.10)
has a unique minimizer v
0
. Further v
0
1(L).
Copyright c _19932011, Andre L. Tits. All Rights Reserved 161
Generalities on Vector Spaces
Proof. We know that L[
1(L

)
is onto 1(L). Let v
0
1(L

) be such that Lv
0
= w. Claim: v
0
is the unique solution to (A.10). Indeed, let v be such that Lv = w. Then v v
0
A(L) =
1(L

. Hence v
0
is the projection of v onto 1(L

). So v
0
, v
0
< v, v unless v = v
0
,
proving the claim.
Corollary A.1 Let V, W be inner-product spaces and let L : V W be linear, with adjoint
L

. Suppose that 1(L) = 1(LL

). Then
V = 1(L

) 1(L

.
Proof. Let v V . Let v
0
1(L

) be such that Lv
0
= Lv. (Such v
0
exists and is unique,
from the previous result.) Then v v
0
A(L) = 1(L

and v = v
0
+ (v v
0
).
Exercise A.47 Prove that 1(L) = 1(LL

) holds if either (i) 1(LL

) is closed, which is
the case, e.g., when W is nite-dimensional, or (ii) V is Hilbert and 1(L

) is closed. See
Figure A.1.
Moore-Penrose Pseudo-inverse
Suppose that 1(L) = 1(LL

). It follows from Exercise A.47 that L[


1(L

)
: 1(L

) 1(L)
is a bijection. Let L

[
1(L)
: 1(L) 1(L

) denote its inverse. Further, dene L

on A(L

)
by
L

w =
V
w A(L

).
Exercise A.48 Suppose W = 1(L) A(L

). (For instance, W is Hilbert and R(L) is


closed.) Prove that L

has a unique linear extension to W.


This extension is the Moore-Penrose pseudo-inverse. L

is linear and LL

restricted to 1(L)
is the identify in W, and L

L restricted to 1(L

) is the identity in V ; i.e.,


LL

L = L and L

LL

= L

. (A.11)
Exercise A.49 Let L : R
n
R
m
be a linear map with matrix representation M
L
(an mn
matrix). We know that the restriction of L to 1(L

) is one-to-one, onto 1(L). Thus there


is an inverse map from 1(L) to 1(L

). The Moore-Penrose pseudo-inverse L

of L is a map
from R
m
to R
n
that agrees with the just mentioned inverse on 1(L) and maps to every
point in A(L

). Prove the following.


Such L

is uniquely dened, i.e., for an arbitrary linear map L, there exists a linear
map, unique among linear maps, that satises all the listed condition.
If L is one-to-one, then L

L is invertible and L

= (L

L)
1
L

.
If L is onto, then LL

is invertible and L

= L

(LL

)
1
.
If L is one-to-one and onto, then L is invertible and L

= L
1
.
Let M
L
= UV
T
be the singular value decomposition of M
L
, and let k be such that the
nonzero entries of are its (i, i) entries, i = 1, . . . , k. Then the matrix representation
of L

is given by M

L
:= V

U
T
where

(an n m matrix) has its (i, i) entry,


i = 1, . . . , k equal to the inverse of that of and all other entries equal to zero.
162 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
Appendix B
On Dierentiability and Convexity
B.1 Dierentiability
[21, 2, 10]
First, let f : R R. We know that f is dierentiable at x

if
lim
t0
f(x

+ t) f(x

)
t
exists, i.e. if there exists a number a such that
f(x

+ t) f(x

)
t
= a + (t)
with (t) 0 as t 0 i.e., if there exists a R such that
f(x

+ t) = f(x

) + at + o(t) t R (B.1)
where
o(t)
t
0 as t 0. The number a is called the derivative at x

and is noted f

(x

).
Obviously, we have
f

(x

) = lim
t0
f(x

+ t) f(x

)
t
.
Equation (B.1) shows that f

(x

)t is a linear approximation to f(x

+ t) f(x

). Our
rst goal in this appendix is to generalize this to f : V W, with V and W more general
(than R) vector spaces. In such context, a formula such as (B.1) (with a = f

(x

)) imposes
that t V and f

(x

)t W. Hence is it natural to stipulate that f

(x

) should be a linear
map from V to W. This leads to Denitions B.3 and B.4 below. But rst we consider the
case of f : R
n
R
m
.
Denition B.1 The function f has partial derivatives at x

if a
ij
R, for i = 1, . . . , m
j = 1, . . . , n such that
f
i
(x

+ te
j
) = f
i
(x

) + a
ij
t + o
ij
(t) t R
with
o
ij
(t)
t
0 as t 0, for i = 1, . . . , m, j = 1, . . . , n. The numbers a
ij
are typically
denoted
f
i
x
j
(x

).
Copyright c _19932011, Andre L. Tits. All Rights Reserved 163
On Dierentiability and Convexity
Example B.1 Let f : R
2
R be dened by
f(x
1
, x
2
) =
_
0 if x
1
= 0 or x
2
= 0
1 elsewhere
Then f has partial derivatives at (0, 0) (their value is 0).
Note that, in the example above, f is not even continuous at (0, 0). Also, the notion of
partial derivative does not readily extend to functions whose domain and co-domain are
not necessarily nite-dimensional. For both of these reasons, we next consider more general
notions of dierentiability. Before doing so, we note the following fact, which applies when
f maps R
n
to R
m
for some n and m.
Fact. (e.g., [10, Theorem 13.20]) Let f : R
n
R
m
, and let be an open subset of R
n
.
If the partial derivatives of (the components of) f exist and are continuous on , then f is
continuous at x.
We now consider f : V W, where V and W are two possibly more general vector
spaces. Suppose W is equipped with a norm.
Denition B.2 f is 1-sided (2-sided) directionally dierentiable at x

V if for all h V
there exists a
h
W such that
f(x

+ th) = f(x

) + ta
h
+ o
h
(t) t R (B.2)
with
1
t
o
h
(t) 0 as t 0 (for any given h) (2 sided)
1
t
o
h
(t) 0 as t 0 (for any given h) (1 sided)
a
h
is the directional derivative of f at x

in direction h.
Denition B.3 f is Gateaux (or G) dierentiable at x

if there exists a linear map


A : V W such that
f(x

+ th) = f(x

) + tAh + o(th) h V t R (B.3)


with
1
t
o(th) 0 as t 0 (for every xed h)
A is the G-derivative of f at x

.
Note: The term G-dierentiability is also used in the literature to refer to other concepts
of dierentiability. Here we follow the terminology used in [21].
If f is G-dierentiable at x

then it is 2-sided directionally dierentiable at x

in all directions,
and the directional derivative in direction h is the image of h under the mapping dened by
the G-derivative.
Suppose now that V is also equipped with a norm.
164 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
B.1 Dierentiability
Denition B.4 f is Frechet (or F) dierentiable at x

if there exists a continuous linear


map A : V W such that
f(x

+ h) = f(x

) + Ah + o(h) h V (B.4)
with
1
|h|
o(h) 0 as h 0 .
In other words, f is F-dierentiable at x* if there exists a continuous linear map A : V W
such that
1
|h|
(f(x

+ h) f(x

) Ah) 0 as h 0 .
If f is F-dierentiable at x

, the corresponding linear map (A) is called Frechet-derivative


of f at x

and is generally noted


f
x
(x

). Using this notation, we can write (B.4) as


f(x

+ h) = f(x

) +
f
x
(x

)h + o(h)
whenever f : V W is F-dierentiable at x

.
If f : R
n
R
m
, then
f
x
(x

) can be represented by a matrix (as any linear map from


R
n
to R
m
).
Exercise B.1 If f : R
n
R
m
, then in the canonical bases for R
n
and R
m
, the entries of
the matrix
f
x
(x

) are the partial derivatives


f
i
x
j
(x

).
It should be clear that, if f is F-dierentiable at x with F-derivative
f
x
(x), then (i) it
is G-dierentiable at x with G-derivative
f
x
(x), (ii) it is 2-sided directionally dierentiable
at x in all directions, with directional derivative in direction h given by
f
x
(x)h, and (iii) if
f : R
n
R
m
, then f has partial derivatives
f
i
x
j
(x) at x equal to
_
f
x
(x)
_
ij
.
The dierence between G ateaux and Frechet is that, for the latter, the limit must be 0 no
matter how v goes to 0, whereas, for the former, convergence is along straight lines v = th.
Further, F-dierentiability at x requires continuity (i.e., boundedness) of the derivative at
x. Clearly, any Frechet-dierentiable function at x is continuous at x (why?).
The following exercises taken from [21] show that each denition is strictly stronger than
the previous one.
Exercise B.2 (proven in [21])
1. If f is Gateaux-dierentiable, its Gateaux derivative is unique.
2. If f is Frechet dierentiable, it is also Gateaux dierentiable and its Frechet derivative
is given by its (unique) Gateaux-derivative.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 165
On Dierentiability and Convexity
Exercise B.3 Dene f : R
2
R by f(x) = x
1
if x
2
= 0, f(x) = x
2
, if x
1
= 0, and
f(x) = 1 otherwise. Show that the partial derivatives
f
x
1
(0) and
f
x
2
(0) exist, but that f is
not directionally dierentiable at 0.
Exercise B.4 Dene f : R
2
R by
f(x) = sgn(x
2
) min([x
1
[, [x
2
[).
Show that, for any h R
2
,
lim
t0
(1/t)[f(th) f(0)] = f(h),
and thus that f is 2-sided directionally dierentiable at 0, but that f is not G-dierentiable
at 0.
Exercise B.5 Dene f : R
2
R by
f(x) =
_

_
0 if x
1
= 0
2x
2
e

1
x
2
1
x
2
2
+
_
e

1
x
2
1
_
2
if x
1
,= 0.
Show that f is G-dierentiable at 0, but that f is not continuous (and thus not F-
dierentiable) at zero.
Remark B.1 For f : R
m
R
n
, again from the equivalence of norms, the various types of
dierentiability do not depend on the particular norm.
Gradient of a dierentiable functional over a Hilbert space
Let f : H R where H is a Hilbert space and suppose f is Frechet-dierentiable at x.
Then, in view of the Riesz-Frechet theorem (see, e.g.,[23, Theorem 4-12]), there exists a
unique g H such that
g, h =
f
x
(x)h h H.
Such g is called the gradient of f at x, and denoted gradf(x), i.e.,
gradf(x), h =
f
x
(x)h h H.
When H = R
n
and , is the Euclidean inner product, we will often denote the gradient
of f at x by f(x

).
Exercise B.6 Note that the gradient depends on the inner product to which it is associated.
For example, suppose f : R
n
R. Let S be a symmetric positive denite nn matrix, and
dene x, y = x
T
Sy. Prove that gradf(x) = S
1 f
x
(x)
T
. In particular, prove that, under
the Euclidean inner product, f(x) =
f
x
(x)
T
.
166 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
B.1 Dierentiability
In the sequel, unless otherwise specied, dierentiable will mean Frechet-dierentiable.
An important property, which may not hold if f is merely G ateaux dierentiable, is given
by the following fact.
Fact [21]. If : X Y and : Y Z are F-dierentiable, respectively at x

X and at
(x

) Y , then h : X Z dened by h(x) = ((x)) is F-dierentiable and the following


chain rule applies

x
h(x

) =

y
(y

)

x
(x

)[
y

= (x

)
( =

y
((x

))

x
(x

))
Exercise B.7 Prove this fact using the notation used in these notes.
Exercise B.8 Compute the total derivative of ((x), (x)) with respect to x, with , ,
and F-dierentiable maps between appropriate spaces.
Exercise B.9 Let Q be an n n (not necessarity symmetric) matrix and let b R
n
. Let
f(x) =
1
2
x, Qx + b, x, where x, y = x
T
Sy, with S = S
T
> 0. Show that f is
F-dierentiable and obtain its gradient with respect to the same inner product.
Remark B.2
1. We will say that a function is dierentiable (in any of the previous senses) if it is
dierentiable everywhere.
2. When x is allowed to move,
f
x
can be viewed as a function of x, whose values are
linear maps
f
x
: V B(V, W), x
f
x
(x)
where B(V, W) is the set of all continuous linear maps from V to W. B(V, W) can be
equipped with a norm, (e.g., the induced norm: if A B(V, W), [[A[[
i
= sup
xV
[[Ax[[
[[x[[
).
Mean Value Theorem (e.g., [21])
First, let : R R and suppose is continuous on [a, b] R and dierentiable on (a, b).
Then, we know that there exists (a, b) such that
(b) (a) =

()(b a) (B.5)
i.e.,
(a + h) (a) =

(a + th)h, for some t (0, 1) (B.6)


We have the following immediate consequence for functionals.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 167
On Dierentiability and Convexity
Fact. Suppose f : V R is dierentiable. Then for any x, h V there exists t (0, 1)
such that
f(x + h) f(x) =
f
x
(x + th)h (B.7)
Proof. Consider : R R dened by (s) : f(x + sh). By the result above there exists
t (0, 1) such that
f(x + h) f(x) = (1) (0) =

(t) =
f
x
(x + th)h (B.8)
where we have applied the chain rule for F-derivatives.
It is important to note that this result is generally not valid for f : V R
m
, m > 1, because
to dierent components of f will correspond dierent values of t. For this reason, we will
often make use, as a substitute, of the fundamental theorem of integral calculus, which
requires continuous dierentiability (though the weaker condition of absolute continuity,
which implies existence of the derivative almost everywhere, is sucient):
Denition B.5 f : V W is said to be continuously Frechet-dierentiable if it is Frechet-
dierentiable and its Frechet derivative
f
x
is a continuous function from V to B(V, W).
The following fact strengthens the result we quoted earlier that, when f maps R
n
to R
m
,
continuity of its partial derivatives implies its own continuity.
Fact. (E.g., [2], Theorem 2.5.) Let f : R
n
R
m
, and let be an open subset of R
n
. If the
partial derivatives of (the components of) f exist on and are continuous at x , then f
is continuously (Frechet) dierentiable at x.
Now, if : R R is continuously dierentiable on [a, b] R, then the fundamental
theorem of integral calculus asserts that
(b) (a) =
_
b
a

()d. (B.9)
Now For a continuously dierentiable g : R
1
R
m
, we dene the integral of g by
_
b
a
g(t)dt =
_

_
_
b
a
g
1
(t)dt
.
.
.
_
b
a
g
m
(t)dt
_

_
(B.10)
For f : V R
m
, we then obtain
Theorem B.1 If f : V R
m
is continuously (Frechet) dierentiable, then for all x, h V
f(x + h) f(x) =
1
_
0
f
x
(x + th)h dt
168 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
B.1 Dierentiability
Proof. Dene : R
1
R
m
by (s) = f(x +sh). Apply (B.9), (B.10) and the chain rule.
Note. For f : V W, W a Banach space, the same result holds, with a suitable denition
of the integral (of integral regulated functions, see [2]).
Corollary B.1 . Let V be a normed vector space. If f : V R
n
is continuously dieren-
tiable, then for all x V
f(x + h) = f(x) + O(|h|)
More precisely, f is locally Lipschitz, i.e., given any x

V there exist > 0, > 0, and


> 0 such that for all h V with |h| , and all x B(x

, ),
|f(x + h) f(x)| |h| . (B.11)
Further, if V is nite-dimensional, then given any > 0, and any r > 0, there exists > 0
such that (B.11) holds for all x B(x

, ) and all h V with |h| r.


Exercise B.10 Prove the above.
A more general version of this result is as follows. (See, e.g, [2], Theorem 2.3.)
Fact. Let V and W be normed vector spaces, and let B be an open ball in V . Let f : V W
be dierentiable on B, and suppose
f
x
(x) is bounded on B. Then there exists > 0 such
that, for all x B, h X such that x + h B,
|f(x + h) f(x)| |h| .
Second derivatives
Denition B.6 Suppose f : V W is dierentiable on V and use the induced norm for
B(V, W). If
f
x
: V B(V, W), x
f
x
(x)
is itself dierentiable, then f is twice dierentiable and the derivative of
f
x
at x V is noted

2
f
x
2
(x) and is called second derivative of f at x. Thus

2
f
x
2
(x) : V B(V, W). (B.12)
and

2
f
x
2
: X B(V, B(V, W)), x

2
f
x
2
(x).
Fact. If f : V W is twice continuously Frechet-dierentiable then its second derivative
is symmetric in the sense that for all u, v V , x V
_

2
f
x
2
(x)u
_
v =
_

2
f
x
2
(x)v
_
u ( W) (B.13)
Copyright c _19932011, Andre L. Tits. All Rights Reserved 169
On Dierentiability and Convexity
[Note: the reader may want to resist the temptation of viewing

2
f
x
2
(x) as a cube matrix.
It is simpler to think about it as an abstract linear map.]
Now let f : H R, where H is a Hilbert space, be twice dierentiable. Then, in
view of the Riess-Frechet Theorem, B(H, R) is isomorphic to H, and in view of (B.12)

2
f
x
2
(x) can be thought of as a map Hessf(x) : H H. This can be made precise as
follows. For any x H,
f
x
(x) : H R is a bounded linear functional and, for any
u H,

2
f
x
2
(x)u : H R is also a bounded linear functional. Thus in view of the Riesz-
Frechet representation Theorem, there exists a unique
u
H such that
_

2
f
x
2
(x)u
_
v =
u
, v v H.
The map from u H to
u
H is linear and bounded (why?). Let us denote this map by

2
f(x) B(H, H). We get
_

2
f
x
2
(x)u
_
v = Hessf(x)u, v.
In view of (B.13), if f is twice continuously dierentiable, Hessf(x) is self-adjoint. If H = R
n
and , is the Euclidean inner product, then Hessf(x) is represented by an nn symmetric
matrix, which we will denote by
2
f(x). In the sequel though, following standard usage, we
will often abuse notation and use
2
f and

2
f
x
2
interchangeably.
Fact. If f : V R
n
is twice F-dierentiable at x V then
f(x + h) = f(x) +
f
x
(x)h +
1
2
_

2
f
x
2
(x)h
_
h + o
2
(h) (B.14)
with
[[o
2
(h)[[
[[h[[
2
0 as h 0
Exercise B.11 Prove the above. Hint: rst use Theorem B.1.
Finally, a second order integral formula.
Theorem B.2 If f : V R
m
is twice continuously dierentiable, then for all x, h V
f(x + h) = f(x) +
f
x
(x)h +
_
1
0
(1 t)
_

2
f
x
2
(x + th)h
_
h dt (B.15)
(This generalizes the relation, with : R R twice cont. di.
(1) (0)

(0) =
_
1
0
(1 t)

(t)dt
Check it by integrating by parts. Let (s) = f(x + s(y x)) to prove the theorem.)
170 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
B.2 Some elements of convex analysis
Corollary B.2 . Let V be a normed vector space. Suppose f : V R
n
is twice continuously
dierentiable, Then given any x

X there exists > 0, > 0, and > 0 such that for all
h X with |h| , and all x B(x

, ),
|f(x + h) f(x)
f
x
(x)h| |h|
2
. (B.16)
Further, if V is nite-dimensional, then given any > 0, and any r > 0, there exists > 0
such that (B.16) holds for all x B(x

, ) and all h V with |h| r.


Again, a more general version of this result holds. (See, e.g, [2], Theorem 4.8.)
Fact. Let V and W be normed vector spaces, and let B be an open ball in W. Let
f : W W be twice dierentiable on B, and suppose

2
f
x
2
(x) is bounded on B. Then there
exists > 0 such that, for all x B, h V such that x + h B,
|f(x + h) f(x)
f
x
(x)h| |h|
2
.
B.2 Some elements of convex analysis
[21]
Let V be a vector space.
Denition B.7 A set S V is said to be convex if, x, y S, [0, 1],
x + (1 )y S. (B.17)
Denition B.8 A function f : V R is said to be convex on the convex set S X if
x, y S, [0, 1],
f(x + (1 )y) f(x) + (1 )f(y). (B.18)
Remark B.3 (B.18) expresses that a function is convex if the arc lies below the chord (see
Figure B.1).
Remark B.4 This denition may fail (in the sense of (B.18) not being well dened) when
f is allowed to take on values of both and +, which is typical in convex analysis. A
more general denition is as follows: : A function f : V R is convex on convex
set S V if its epigraph
epi f := (x, z) S R : z f(x)
is a convex set.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 171
On Dierentiability and Convexity
f
x y
f(y)+(f(x)-f(y))
=f(x)+(1-)f(y)
f(x+(1-)y)
x+(1-)y
=y+(x-y)

Figure B.1:
Fact. If f : R
n
R is convex, then it is continuous. More generally, if f : V R
is nowhere equal to , then it is continuous over the relative interior of its domain
(hence over the interior of its domain). See, e.g., [5], section 1.4 and [21].
Denition B.9 A function f : V R is said to be strictly convex on the convex set S V
if x, y S, x ,= y, (0, 1),
f(x + (1 )y) < f(x) + (1 )f(y). (B.19)
Denition B.10 A convex combination of k points x
1
, . . . , x
k
V is any point x expressible
as
x =
k

i=1

i
x
i
(B.20)
with
i
0, for i = 1, . . . , k and
k

i=1

i
= 1.
Exercise B.12 Show that a set S is convex if and only if it contains the convex combinations
of all its nite subsets. Hint: use induction.
Fact 1. [21]. A function f : V R is convex on the convex set S V , if and only if for
any nite set of points x
1
, ..., x
k
S and any
i
0, i = 1, ..., k with
k

i=1

i
= 1, one has
f(
k

i=1

i
x
i
)
k

i=1

i
f(x
i
).
Exercise B.13 Prove the above. (Hint: Use Exercise B.12 and mathematical induction.)
172 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
B.2 Some elements of convex analysis
If the domain of f is R
n
, convexity implies continuity.
Convex Hull
Denition B.11 The convex hull of a set X, denoted coX, is the smallest convex set con-
taining X. The following exercise shows that it makes sense to talk about the smallest
such set.
Exercise B.14 Show that Y : Y convex, X Y is convex and contains X. Since it is
contained in any convex set containing X, it is the smallest such set.
Exercise B.15 Prove that
coX =

_
k=1

i=1

i
x
i
: x
i
X,
i
0 i,
k

i=1

i
= 1
In other words, coX is the set of all convex combinations of nite subsets of X. In particular,
if X is nite, X = x
1
, . . . , x

coX =

i=1

i
x
i
:
i
0 i,

i=1

i
= 1
(Hint: To prove , show that X RHS (right-hand side) and that RHS is convex. To
prove , show using mathematical induction that if x RHS, then x = x
1
+(1)x
2
with
[0, 1] and x
1
, x
2
coX.)
Exercise B.16 (Caratheodory; e.g. [9]). Show that, in the above exercise, for X R
n
, it
is enough to consider convex combinations of n + 1 points, i.e.,
coX =
n+1

i=1

i
x
i
:
i
0, x
i
X,
n+1

i=1

i
= 1
and show by example (say, in R
2
) that n points is generally not enough.
Exercise B.17 Prove that the convex hull of a compact subset of R
n
is compact and that
the closure of a convex set is convex. Show by example that the convex hull of a closed subset
of R
n
need not be closed.
Suppose now V is a normed space.
Proposition B.1 Suppose that f : V R is dierentiable on a convex subset S of V .
Then f : V R is convex on S if and only if, for all x, y S
f(y) f(x) +
f
x
(x)(y x). (B.21)
Copyright c _19932011, Andre L. Tits. All Rights Reserved 173
On Dierentiability and Convexity
Proof. (only if) (see Figure B.2). Suppose f is convex. Then, x, y V, [0, 1],
f(x + (y x)) = f(y + (1 )x) f(y) + (1 )f(x) = f(x) + (f(y) f(x))(B.22)
i.e.
f(y) f(x)
f(x + (y x)) f(x)

(0, 1], x, y V (B.23)


and, when 0, since f is dierentiable,
f(y) f(x)
f
x
(x)(y x) x, y V (B.24)
(if) (see Figure B.3). Suppose (B.21) holds x, y V . Then, for given x, y V and
z = x + (1 )y
f(x) f(z) +
f
x
(z)(x z) (B.25)
f(y) f(z) +
f
x
(z)(y z) (B.26)
(B.25) +(1 ) (B.26) yields
f(x) + (1 )f(y) f(z) +
f
x
(z)(x + (1 )y z)
= f(x + (1 )y)
Fact. Moreover, f is strictly convex if and only if inequality (B.21) is strict whenever x ,= y
(see [21]).
Denition B.12 f : V R is said to be strongly convex over a convex set S if f is
continuously dierentiable on S and there exists m > 0 s.t. x, y S
f(y) f(x) +
f
x
(x)(y x) +
m
2
|y x|
2
.
We now focus on the case V = R
n
. This simplies the notation. Also, some of the results
do not hold in the general case.
Proposition B.2 If f : R
n
R is strongly convex, it is strictly convex and, for any
x
0
R
n
, the sublevel set
x[f(x) f(x
0
)
is bounded.
174 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
B.2 Some elements of convex analysis
Proof. The rst claim follows from Exercise B.19. Now, let h be an arbitrary unit vector in
R
n
. Then
f(x
0
+ h) f(x
0
) +
f
x
(x
0
)h +
1
2
m|h|
2
f(x
0
) |
f
x
(x
0
)||h| +
1
2
m|h|
2
= f(x
0
) +
_
m
2
|h| |
f
x
(x
0
)|
_
|h|
> f(x
0
) whenever |h| > (2/m)|
f
x
(x
0
)|.
Hence x[f(x) f(x
0
) B
_
x
0
, (2/m)|
f
x
(x
0
)|
_
, which is a bounded set.
Interpretation. The function f lies above the function

f(x) = f(x
0
) +
f
x
(x
0
)(x x
0
) +
1
2
m|x x
0
|
2
which grows with bound uniformly in all directions.
Proposition B.3 Suppose that f : R
n
R is twice continuously dierentiable. Then f is
convex if and only if the Hessian
2
f(x) is positive semi-denite for all x R
n
.
Proof. We prove suciency. We can write, x, y R
n
f(x + h) = f(x) +
f
x
(x)h +
_
1
0
(1 t)h
T

2
f(x + th)hdt
f(y) f(x) +
f
x
(x)h x, y R
n
and, from previous proposition, f is convex.
Exercise B.18 Prove the necessity part of Proposition B.3.
Exercise B.19 Show that if
2
f is positive denite on R
n
, then f is strictly convex. Show
by example that the converse does not hold in general.
Exercise B.20 Suppose f : R
n
R is twice continuously dierentiable. Then f is strongly
convex if and only there exists m > 0 such that for all x, h R
n
,
h
T

2
f(x)h m[[h[[
2
.
Exercise B.21 Show that the above holds if, and only if, the eigenvalues of
2
f(x) (they
are real, why ?) are all positive and bounded away from zero, i.e., there exists m > 0 such
that for all x R
n
all eigenvalues of
2
f(x) are larger than m.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 175
On Dierentiability and Convexity
which, due to a constant positive denite Hessian, grows unbounded uniformly in all direc-
tions.
Exercise B.22 Exhibit a function f : R R, twice continuously dierentiable with Hes-
sian everywhere positive denite, which is not strongly convex.
Exercise B.23 Exhibit a function f : R
2
R such that for all x, y R, f(x, ) : R R
and f(, y) : R R are strongly convex but f is not even convex. Hint: consider the Hessian
matrix.
Separation of Convex Sets [21, 19]
For simplicy, we focus on R
n
.
Denition B.13 Let a R
n
, a ,= 0, and R. The set
P(a, ) = x R
n
: a
T
x =
is called a hyperplane.
Let us check, for n = 2, that this corresponds to the intuitive notion we have of a hyperplane,
i.e., in this case, a straight line can be expressed as x = a+v for some va (see Figure B.4).
Hence
a, x = [[a[[
2
+a, v = [[a[[
2
z H
which corresponds to the above denition with = [[a[[
2
.
x : a
T
x and x : a
T
x are closed half spaces;
x : a
T
x < and x : a
T
x > are open half spaces.
Denition B.14 Let X, Y R
n
. X and Y are (strictly) separated by P(a, ) if
a
T
x (>) x X
a
T
y (<) y Y
This means that X is entirely in one of the half spaces and Y entirely in the other. Examples
are given in Figure B.5.
Note. X and Y can be separated without being disjoint. Any hyperplane is even separated
from itself (by itself!)
The idea of separability is strongly related to that of convexity. For instance, it can be shown
that 2 disjoint convex sets can be separated. We will present here only one of the numerous
separation theorems; this theorem will be used in connection with constrained optimization.
Theorem B.3 Suppose X R
n
is nonempty, closed and convex and suppose that 0 , X.
Then a R
n
, a ,= 0, and > 0 such that
a
T
x > x X (B.27)
(i.e., X and 0 are strictly separated by P(a, )).
176 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
B.2 Some elements of convex analysis
f(x)+(f(y)-f(x))
f(y)
f(x) + f(x+(y-x))-f(x)

f(x+(y-x))
x
0
y
1

Figure B.2:
x
A
y
1
z
A
C B
B
C
1

C above C
1 A above A
B above B
1
1
Figure B.3:
x
H
1a
a
a+x

Figure B.4:
Copyright c _19932011, Andre L. Tits. All Rights Reserved 177
On Dierentiability and Convexity
Proof (see Figure B.6). We rst show that H
1
separates 0 and X, then that H
2
strictly
separates 0 and X. First choose > 0 such that

B(0, ) X ,= .

B(0, ) X is bounded
and closed (intersection of closed sets), hence compact. Since [[ [[ is continuous, x X
[[ x[[ = min[[x[[ [ x

B(0, ) X (B.28)
i.e.
[[ x[[ [[x[[ x

B(0, ) X (B.29)
hence
[[ x[[ [[x[[ x X (B.30)
(since [[ x[[ and [[x[[ > x ,

B(0, )) and x is the closest point to the origin in X.
Since x X, [[ x[[ , = 0. Now let | | denote the Euclidean norm. We show that H
1
separates
X from 0, i.e., for all x X
x
T
x [[ x[[
2
. (B.31)
By contradiction: suppose there exists x X s.t.
x
T
x = [[ x[[
2
, > 0. (B.32)
Since X is convex,
x

= x + (1 ) x = x + (x x) X [0, 1].
We show that, for small ,
[[x

[[
2
< [[ x[[
2
(B.33)
which contradicts (B.30). We have
H
x
H
y
x
y
Figure B.5:
X and Y are separated by H and strictly separated by H

.
X

and Y

are not separated by any hyperplane (i.e., cannot be separated).


178 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
B.2 Some elements of convex analysis

x, x =x = H

1
2

x, x = x = H

2
2
2
1
x
X
x
0
Figure B.6:
[[x

[[
2
= [[ x[[
2
+
2
[[x x[[
2
+ 2( x, x [[ x[[
2
) [0, 1]
i.e. using (B.32)
[[x

[[
2
= [[ x[[
2
+
2
[[x x[[
2
2
and (B.33) holds for small enough (since > 0).
Hence, (B.31) must hold. It is now easy to show that H
2
strictly separates X and 0. Since
0 , X, [[ x[[ , = 0 and
| x|
2
2
< | x|
2
.
Hence x
T
x >
| x|
2
2
for all x X and this proves the theorem.
Fact. If X and Y are nonempty disjoint and convex, with X compact and Y closed, then
X and Y are strictly separated.
Exercise B.24 Prove the Fact. Hint: rst show that Y X, dened as z[z = y x, y
Y, x X, is closed, convex and does not contain 0.
Exercise B.25 Show by an example that if in the above theorem, X is merely closed, X
and Y may not be strictly separated. (In particular, the dierence of 2 closed sets, dened
as above, may not be closed.)
Copyright c _19932011, Andre L. Tits. All Rights Reserved 179
On Dierentiability and Convexity
B.3 Acknowledgment
The author wishes to thank the numerous students who have contributed constructive com-
ments towards improving these notes over the years. In addition, special thanks are addressed
to Ji-Woong Lee, who used these notes when he taught an optimal control course at Penn
State University in the Spring 2008 semester and, after the semester was over, provided
many helpful comments towards improving the notes.
180 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved
Bibliography
[1] B.D.O. Anderson and J.B. Moore. Optimal Control: Linear Quadratic Methods. Prentice
Hall, 1990.
[2] A. Avez. Dierential Calculus. J. Wiley and Sons, 1986.
[3] D.P. Bertsekas. Constrained Optimization and Lagrange Multiplier Methods. Academic
Press, New York, 1982.
[4] S.P. Bertsekas. Nonlinear Programming. Athena Scientic, Belmont, Massachusetts,
1996.
[5] S.P. Bertsekas, A. Nedic, and A.E. Ozdaglar. Convex Analysis and Optimization. Athena
Scientic, Belmont, Massachusetts, 2003.
[6] F.M. Callier and C.A. Desoer. Linear System Theory. Springer-Verlag, New York, 1991.
[7] M.D. Canon, JR. C.D. Cullum, and E. Polak. Theory of Optimal Control and Mathe-
matical Programming. McGRAW-HILL BOOK COMPANY, New York, 1970.
[8] F.H. Clarke. Optimization and Nonsmooth Analysis. Wiley Interscience, 1983.
[9] V.F. Demyanov and L.V. Vasilev. Nondierentiable Optimization. Translations Series
in Mathematics and Engineering. Springer-Verlag, New York, Berlin, Heidelberg, Tokyo,
1985.
[10] P.M. Fitzpatrick. Advanced Calculus. Thomson, 2006.
[11] W.H. Fleming and R.W. Rishel. Deterministic and Stochastic Optimal Control.
Springer-Verlag, New York, 1975.
[12] J.K. Hale. Ordinary Dierential Equations. Wiley-Interscience, 1969.
[13] H.K. Khalil. Nonlinear Systems. Prentice Hall, 2002. Third Edition.
[14] H. Kwakernaak and S. Sivan. Linear Optimal Control Systems. Wiley, Interscience,
1972.
[15] Peter D. Lax. Functional Analysis. J. Wiley & Sons Inc., 2002.
[16] E.B. Lee and L. Markus. Foundations of Optimal Control Theory. Wiley, New York,
1967.
Copyright c _19932011, Andre L. Tits. All Rights Reserved 181
BIBLIOGRAPHY
[17] G. Leitman. The Calculus of Variations and OptimalControl. Plenum Press, 1981.
[18] D. G. Luenberger. Introduction to Linear and Nonlinear Programming. Addison-Wesley,
Reading, Mass., 1973.
[19] D.G. Luenberger. Optimization by Vector Space Methods. J. Wiley and Sons, 1969.
[20] J. Nocedal and S.J. Wright. Numerical Optimization. Second edition. Springer-Verlag,
2006.
[21] J. Ortega and W. Rheinboldt. Iterative Solution of Nonlinear Equations in Several
Variables. Academic Press, New York, 1970.
[22] E. Polak. Computational Methods in Optimization. Academic Press, New York, N.Y.,
1971.
[23] W. Rudin. Real and Complex Analysis. McGraw-Hill, New York, N.Y., 1974. second
edition.
[24] W.J. Rugh. Linear System Theory. Prentice Hall, 1993.
[25] H.J. Sussmann and J.C. Willems. 300 years of optimal control: From the brachys-
trochrone to the maximum pricniple. IEEE CSS Magazine, 17, 1997.
[26] P.P. Varaiya. Notes in Optimization. Van Nostrand Reinhold, 1972.
[27] K. Zhou, J.C. Doyle, and K. Glover. Robust and Optimal Control. Prentice Hall, New
Jersey, 1996.
182 Copyright c _1993-2011, Andre L. Tits. All Rights Reserved

You might also like