Professional Documents
Culture Documents
5. Duality
5–1
Lagrangian
minimize f0(x)
subject to fi(x) ≤ 0, i = 1, . . . , m
hi(x) = 0, i = 1, . . . , p
Duality 5–2
Lagrange dual function
Duality 5–3
minimize xT x
subject to Ax = b
dual function
• Lagrangian is L(x, ν) = xT x + ν T (Ax − b)
• to minimize L over x, set gradient equal to zero:
∇xL(x, ν) = 2x + AT ν = 0 =⇒ x = −(1/2)AT ν
• plug in in L to obtain g:
1
g(ν) = L((−1/2)AT ν, ν) = − ν T AAT ν − bT ν
4
a concave function of ν
Duality 5–4
Standard form LP
minimize cT x
subject to Ax = b, x0
dual function
• Lagrangian is
L(x, λ, ν) = cT x + ν T (Ax − b) − λT x
= −bT ν + (c + AT ν − λ)T x
• L is linear in x, hence
−bT ν AT ν − λ + c = 0
g(λ, ν) = inf L(x, λ, ν) =
x −∞ otherwise
Duality 5–5
minimize kxk
subject to Ax = b
dual function
bT ν kAT νk∗ ≤ 1
T T
g(ν) = inf (kxk − ν Ax + b ν) =
x −∞ otherwise
where kvk∗ = supkuk≤1 uT v is dual norm of k · k
Duality 5–6
Two-way partitioning
minimize xT W x
subject to x2i = 1, i = 1, . . . , n
dual function
X
g(ν) = inf (xT W x + νi(x2i − 1)) = inf xT (W + diag(ν))x − 1T ν
x x
i
−1T ν W + diag(ν) 0
=
−∞ otherwise
Duality 5–7
minimize f0(x)
subject to Ax b, Cx = d
dual function
= −f0∗(−AT λ − C T ν) − bT λ − dT ν
Duality 5–8
The dual problem
maximize g(λ, ν)
subject to λ 0
• finds best lower bound on p⋆, obtained from Lagrange dual function
• a convex optimization problem; optimal value denoted d⋆
• λ, ν are dual feasible if λ 0, (λ, ν) ∈ dom g
• often simplified by making implicit constraint (λ, ν) ∈ dom g explicit
Duality 5–9
maximize −1T ν
subject to W + diag(ν) 0
gives a lower bound for the two-way partitioning problem on page 5–7
strong duality: d⋆ = p⋆
• does not hold in general
• (usually) holds for convex problems
• conditions that guarantee strong duality in convex problems are called
constraint qualifications
Duality 5–10
Slater’s constraint qualification
minimize f0(x)
subject to fi(x) ≤ 0, i = 1, . . . , m
Ax = b
• also guarantees that the dual optimum is attained (if p⋆ > −∞)
• can be sharpened: e.g., can replace int D with relint D (interior
relative to affine hull); linear inequalities do not need to hold with strict
inequality, . . .
• there exist many other types of constraint qualifications
Duality 5–11
Inequality form LP
primal problem
minimize cT x
subject to Ax b
dual function
−bT λ AT λ + c = 0
T T T
g(λ) = inf (c + A λ) x − b λ =
x −∞ otherwise
dual problem
maximize −bT λ
subject to AT λ + c = 0, λ0
Duality 5–12
Quadratic program
primal problem (assume P ∈ Sn++)
minimize xT P x
subject to Ax b
dual function
1
g(λ) = inf xT P x + λT (Ax − b) = − λT AP −1AT λ − bT λ
x 4
dual problem
maximize −(1/4)λT AP −1AT λ − bT λ
subject to λ 0
Duality 5–13
minimize xT Ax + 2bT x
subject to xT x ≤ 1
A 6 0, hence nonconvex
strong duality although primal problem is not convex (not easy to show)
Duality 5–14
Geometric interpretation
for simplicity, consider problem with one constraint f1(x) ≤ 0
interpretation of dual function:
t t
G G
p⋆ p⋆
λu + t = g(λ) d⋆
g(λ)
u u
Duality 5–15
λu + t = g(λ) p⋆
g(λ)
u
strong duality
• holds if there is a non-vertical supporting hyperplane to A at (0, p⋆)
• for convex problem, A is convex, hence has supp. hyperplane at (0, p⋆)
• Slater’s condition: if there exist (ũ, t̃) ∈ A with ũ < 0, then supporting
hyperplanes at (0, p⋆) must be non-vertical
Duality 5–16
Complementary slackness
Duality 5–17
the following four conditions are called KKT conditions (for a problem with
differentiable fi, hi):
m
X p
X
∇f0(x) + λi∇fi(x) + νi∇hi(x) = 0
i=1 i=1
from page 5–17: if strong duality holds and x, λ, ν are optimal, then they
must satisfy the KKT conditions
Duality 5–18
KKT conditions for convex problem
if x̃, λ̃, ν̃ satisfy KKT for a convex problem, then they are optimal:
• recall that Slater implies strong duality, and dual optimum is attained
• generalizes optimality condition ∇f0(x) = 0 for unconstrained problem
Duality 5–19
interpretation
• n patches; level of patch i is at height αi
1/ν ⋆
xi
• flood area with unit amount of water
αi
• resulting level is 1/ν ⋆
i
Duality 5–20
Perturbation and sensitivity analysis
(unperturbed) optimization problem and its dual
Duality 5–21
p⋆(u, v) ≥ g(λ⋆, ν ⋆) − uT λ⋆ − v T ν ⋆
= p⋆(0, 0) − uT λ⋆ − v T ν ⋆
sensitivity interpretation
Duality 5–22
local sensitivity: if (in addition) p⋆(u, v) is differentiable at (0, 0), then
∂p⋆(0, 0) ∂p⋆(0, 0)
λ⋆i =− , νi⋆ =−
∂ui ∂vi
p⋆(0) − λ⋆u
Duality 5–23
common reformulations
Duality 5–24
Introducing new variables and equality constraints
minimize f0(Ax + b)
−f0∗(ν) + bT ν AT ν = 0
=
−∞ otherwise
Duality 5–25
minimize kyk
subject to y = Ax − b
maximize bT ν
subject to AT ν = 0, kνk∗ ≤ 1
Duality 5–26
Implicit constraints
LP with box constraints: primal and dual problem
dual function
g(ν) = inf (cT x + ν T (Ax − b))
−1x1
Duality 5–27
minimize f0(x)
subject to fi(x) Ki 0, i = 1, . . . , m
hi(x) = 0, i = 1, . . . , p
Duality 5–28
lower bound property: if λi Ki∗ 0, then g(λ1, . . . , λm, ν) ≤ p⋆
proof: if x̃ is feasible and λ Ki∗ 0, then
m
X p
X
f0(x̃) ≥ f0(x̃) + λTi fi(x̃) + νihi(x̃)
i=1 i=1
≥ inf L(x, λ1, . . . , λm, ν)
x∈D
= g(λ1, . . . , λm, ν)
Duality 5–29
Semidefinite program
primal SDP (Fi, G ∈ Sk )
minimize cT x
subject to x1F1 + · · · + xnFn G
dual SDP
maximize − tr(GZ)
subject to Z 0, tr(FiZ) + ci = 0, i = 1, . . . , n
Duality 5–30