1 Lagrangian Methods

1.1 Lagrangian methods

Let P(b) denote the optimization problem minimize f(x) , subject to h(x)
= b, x X . Let x X(b) = {x X : h(x) = b}. We say that x is feasible if x
X(b).
Define the Lagrangian as
L(x, ) = f(x) (h(x) b). Typically, X Rn, h: Rn 7 Rm, with b,
Rm. Here is called a Lagrangian multiplier.
Theorem 1.1 (Lagrangian Sufficiency Theorem) If x is feasible for P (b)
and there exists such that inf xX L(x,) = L(x, ) then x is optimal
for P(b).
Proof of the LST. For all x X (b) and we have
F (x) = f(x) (h(x) b) = L(x, ) .
Now x X (b) X and so by assumption,
f(x) = L(x, ) L(x, ) = f(x) , for all x X(b) .
Thus x is optimal for the optimization problem P (b).
Example 1.1 Minimize x21
+ x22 subject to a1x1 + a2x2 = b, x1, x2 0.
Here L = x21 + x22 (a1x1 + a2x2 b). We consider the problem
minimize x1, x 20 [x21 + x22 (a1x1 + a2x2 b)].
This has a stationary point where (x1, x2) = (a1/2, a2/2). Now we
choose such that a1x1 + a2x2 = b. This happens for = 2b/ (a21+ a22).
We have a minimum since 2L/x2i> 0, 2L/x1x2 = 0. Thus with this
value of the conditions of the LST are satisfied and the optimal value is
b2/ (a21+ a22) at (x 1, x2) = (a1b, a2b)/(a21+ a22).1
The Dual Problem 2
1.2 The Dual Problem
Let us define (b) = inf xX(b) f(x) and g() = infxX
L (x, ). Then for all (b) = inf xX(b) L(x, ) inf xX
L(x, ) = g () . (1.1)
Thus g () is a lower bound on (b), i.e., a lower bound on the solution
value of P(b). As this holds for all it is interesting to make this lower
bound as large as possible. Of course we can restrict ourselves to for
which g () > . This motivates the dual problem, defined as maximize
g() , subject to Y ,where Y = { : g() > }. In (1.1) we have a proof
of the so-called weak duality theorem that inf xX(b) f(x) max Y g() .
(1.2)
The left hand side of (1.2) poses the primal problem.
1.3 Strong Lagrangian
We say that P(b) is Strong Lagrangian if there exists such that
(b) = inf
xX
L(x, ) . (1.3)
In other words, P(b) is Strong Lagrangian if it can be solved by the
Lagrangian method. But when does this happen? Usually we just try the
method and see. If we are lucky, as in Example 1.1, then we will be able
to establish that there exists a that lets us solve P(b) this way.
However, there are important classes of problems for which we can
guarantee that Lagrangian methods always work.
Note that (1.3) says that there is a such that (b) = g(). Combining this
with (1.2), we see that if the problem is Strong Lagrangian then min of
primal = max of dual.
1.4 Hyperplanes
Let the hyperplane (c, ) be given by
= (b c) .
3 Lagrangian Methods
It has intercept at on the vertical axis through b, and has slope(s) .
Consider
The following approach to finding (b):
1. For each , find max such that the hyperplane lies completely
below the graph of .
2. Now choose to maximize . (c) (c)
Case 1 Case 2
= (b) < (b) b b
Lagrangian methods work in Case 1 because of the existence of a
tangent to at b. Define a supporting hyperplane (c, ) at b as
= (b) (b c) , where (c) (b) (b c) for all c Rm .
In fact, = g() = minxX L(x, ). To see this, we argue g() = inf xX L(x,
) = inf cRm inf xX(c) [f(x) (h(x) b)] = inf cRm [(c) (c b)] =
sup{ : (b c) (c) , for all c Rm} = .
Hence, the dual problem is max . Again, we see the weak duality
result of max (b), with equality if the problem is Strong
Lagrangian.
Theorem 1.2 The following are equivalent:
(a) There exists a (non-vertical) supporting hyperplane to at b;
(b) The problem is Strong Lagrangian.
Hyperplanes 4
This is important since a (non-vertical) supporting hyperplane exists if
(b) is a convex function of b. We can find conditions that make
convex.
Proof. Suppose there exists a (non-vertical) supporting hyperplane to
at b.
This means that there exists such that (b) (b c) (c) for all c
Rm .
This implies (b) inf cRm _ (c) (c b) _ = inf cRm
Inf xX(c) _ f(x) (h(x) b) _ = inf xX L(x, ) = g()
However, we have the opposite inequality in (1.1). Hence (b) = g().
This means that P(b) is Strong Lagrangian, i.e., can be solved by
minimizing L(x, ) with respect to x. Conversely, if the problem is Strong
Lagrangian then there exists such that for all x X (b) f(x)
(h(x) b) . Imagine minimizing the right hand side over x X(c), where
h(x) = c. This gives (b) (c) (c b). This is true for all c, and hence
(b) (b c) (c) for all c Rm.
Hence, has a (non-vertical) supporting hyper plane at b.
A.2 the Lagrangian method
The solution of a constrained optimization problem can often be found
by using the so-called
Lagrangian method. We define the Lagrangian as
L(x, ) = f(x) + (b g(x)) .331
A.2 The Lagrangian method 332
For P1 it is L1(x, ) = _n i=1 wi log xi + _b _n i=1 xi _ .
In general, the Lagrangian is the sum of the original objective function
and a term that involves the functional constraint and a Lagrange
multiplier . Suppose we ignore the functional constraint and consider
the problem of maximizing the Lagrangian, subject only to the regional
constraint. This is often an easier problem than the original one. The
value of x that maximizes L(x, ) depends on the value of . Let us
denote this optimizing value of x by x().
For example, since L1(x, ) is a concave function of x it has a unique
maximum at a point where f is stationary with respect to changes in x,
i.e., where L1/xi = wi/xi = 0 for all i. Thus xi () = wi/. Note that
xi() > 0 for > 0, and so the solution lies in the interior of the feasible
set. Think of as knob that we can turn to adjust the value of x. Imagine
turning this knob until we find a value of , say = , such that the
functional constraint is satisfied, i.e., g(x()) = b. Let x = x(). Our
claim is that x solves P. This is the so-called Lagrangian Sufficiency
Theorem, which we state and prove shortly. First note that, in our
example, g(x()) =_i wi/.
Thus choosing =_i wi/b, we have g(x()) =b. The next theorem shows
that x = x() = wib/_ j wj is optimal for P1.