1

2011 The MathWorks, Inc.

Global Optimization with MATLAB
Products
Account Manager

Application Engineer

2
Agenda
Introduction to Global Optimization

Survey of Solvers with Examples
MultiStart
Global Search
Pattern Search
Simulated Annealing
Genetic Algorithm / Multiobjective
Genetic Algorithm

Additional Resources

Question & Answer
-2
0
2
-3
-2
-1
0
1
2
3
-6
-4
-2
0
2
4
6
8
x
Peaks
y
Local minima
Global minima
3
Optimization Finding answers to problems automatically
Objectives
Achieved?
NO
Optimal
Design
YES
Model or
Prototype
Modify Design
Parameters
Initial
Design
Parameters
OPTIMIZATION PROCESS
Finding better (optimal) designs
Faster design evaluations
Useful for trade-off analysis (N dimensions)
Non-intuitive designs may be found

Optimization benefits include: Design process can be performed:
Antenna Design Using Genetic Algorithm
http://ic.arc.nasa.gov/projects/esg/research/antenna.htm
Manually
(trial-and-error or iteratively)
Automatically
(using optimization techniques)
4
0 20 40 60 80 100 120
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
t
c
c=b
1
e
-b
4
t
+b
2
e
-b
5
t
+b
3
e
-b
6
t
Example Global Optimization Problems
Why does fmincon have a hard time finding the
function minimum?
Why didnt nonlinear regression find a good fit?
Why didnt fminunc find the maximum volumetric
efficiency?
0 5 10
-10
-5
0
5
10
x
Starting at 10
0 5 10
-10
-5
0
5
10
x
Starting at 8
0 5 10
-10
-5
0
5
10
x
Starting at 6
x

s
i
n
(
x
)

+

x

c
o
s
(
2

x
)
0 5 10
-10
-5
0
5
10
x
Starting at 3
0 5 10
-10
-5
0
5
10
x
Starting at 1
0 5 10
-10
-5
0
5
10
x
Starting at 0
x

s
i
n
(
x
)

+

x

c
o
s
(
2

x
)
Revolutions Per Minute, RPM
M
a
n
i
f
o
l
d

P
r
e
s
s
u
r
e

R
a
t
i
o
Peak VE Value = 0.96144


Start
End
1000 2000 3000 4000 5000 6000
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
5
Global Optimization
Goal:
Want to find the lowest/largest value of
the nonlinear function that has many local
minima/maxima

Problem:
Traditional solvers often return one of the
local minima (not the global)

Solution:
A solver that locates globally optimal
solutions
Rastrigins Function
6
Global Optimization Toolbox For problems that
contain multiple maxima/minima or are non-smooth
Optimization T
oolbox
Global Optimization
Toolbox
Faster/fewer function eval
uations

Larger problems (higher d
imensions)

Finds local minima/maxim
a
Finds global minima/maxi
ma (most of the time)

Better on
non-smooth
stochastic
discontinuous
undefined gradients


Custom data types
(in GA and SA solvers)

7
MULTISTART
8
What is MultiStart?
Run a local solver from
each set of start points

Option to filter starting
points based feasibility

Supports parallel
computing


11
GLOBAL SEARCH
12
What is GlobalSearch?
Multistart heuristic algorithm

Calls fmincon from multiple
start points to try and find a
global minimum

Filters/removes non-promising
start points

13
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
GlobalSearch Overview
Schematic Problem
Peaks function
Three minima
Green, z = -0.065
Red, z= -3.05
Blue, z = -6.55
x
y
14
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
GlobalSearch Overview Stage 0
Run from specified x0
x
y
15
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
GlobalSearch Overview Stage 1
Generate stage 1 start points via Scatter Search
3
6
0
0 0
4
0
-2
x
y
16
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
GlobalSearch Overview Stage 1
Find stage 1 start point with lowest penalty value
3
6
0
0 0
4
0
-2
x
y
17
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
GlobalSearch Overview Stage 1
Run from best stage 1 point
x
y
18
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
GlobalSearch Overview Stage 2
Generate stage 2 start points using Scatter Search
x
y
19
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
GlobalSearch Overview Stage 2
Analyse each stage 2 point in turn.
x
y
20
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
GlobalSearch Overview Stage 2
Dont run points that are in basins of existing
minimum
x
y
21
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
GlobalSearch Overview Stage 2
Analyse each stage 2 point in turn.
x
y
22
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
GlobalSearch Overview Stage 2
Dont run points whose penalty value exceeds
threshold
6
Current penalty
threshold value : 4
x
y
23
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
GlobalSearch Overview Stage 2
Analyse each stage 2 point in turn
x
y
24
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
GlobalSearch Overview Stage 2
Run start point if it satisfies distance & merit criteria
Current penalty
threshold value : 4
-3
x
y
25
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
GlobalSearch Overview Stage 2
Expand basin of attraction if minimum already found
Current penalty threshold value : 2
-0.1
x
y Basins can overlap
28
SIMULATED ANNEALING
29
What is Simulated Annealing?
A probabilistic metaheuristic approach based upon the
physical process of annealing in metallurgy.

Controlled cooling of a metal allows atoms to realign from a
random higher energy state to an ordered crystalline
(globally) lower energy state

By analogy, simulated annealing replaces a current solution
by randomly choosing a nearby solution

A nearby solution is determined by the solution
temperature


30
Simulated Annealing Overview Iteration 1
Run from specified x0
x
y
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
0.9
31
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Simulated Annealing Overview Iteration 1
Randomly generate a new point according to probability distribution and
current temperature
3
x
y
0.9
Possible New Points:
Standard Normal N(0,1) * Temperature
Temperature = 1
32
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Simulated Annealing Overview Iteration 1
If lower, accept the point, if higher, accept based upon acceptance
probability
3
x
y
0.9
Temperature = 1
11 . 0
1
1
/ ) 9 . 0 3 (

T accept
e
P
33
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Simulated Annealing Overview Iteration 1
Randomly generate a new point according to probability distribution and
current temperature
3
x
y
0.9
Temperature = 1
0.3
34
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Simulated Annealing Overview Iteration 1
Randomly generate a new point according to probability distribution and
current temperature, accept new point if lower value
3
x
y
0.9
Temperature = 1
0.3
35
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Simulated Annealing Overview Iteration 2
Lower temperature according to temperature schedule
3
x
y
0.9
Temperature = 1
0.3
36
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Simulated Annealing Overview Iteration 2
Lower temperature according to temperature schedule and generate new
point
3
x
y
0.9
Temperature = 0.75
0.3
-1.2
37
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Simulated Annealing Overview Iteration N-1
After several iterations, the search radius becomes small and we narrow
in on a local solution
3
x
y
0.9
Temperature = 0.1
0.3
-1.2
-3
38
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Simulated Annealing Overview Iteration N
Reset temperature and start the process again (reanneling)
3
x
y
0.9
Temperature = 1
0.3
-1.2
-3
-2
39
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Simulated Annealing Overview Iteration N
Reset temperature and start the process again (reannealing)
3
x
y
0.9
Temperature = 1
0.3
-1.2
-3
-2
27 . 0
1
1
/ )) 3 ( 2 (

T accept
e
P
40
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Simulated Annealing Overview Iteration N
Reset temperature and start the process again (reannealing)
3
x
y
0.9
Temperature = 1
0.3
-1.2
-3
-2
41
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Simulated Annealing Overview Iteration N+1
Reduce temperature and continue
3
x
y
0.9
Temperature = 0.75
0.3
-1.2
-3
-2
-3
42
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Simulated Annealing Overview Iteration N+1
Reduce temperature and continue
3
x
y
0.9
Temperature = 0.75
0.3
-1.2
-3
-2
-3
43
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Simulated Annealing Overview Iteration
Reduce temperature and continue
3
x
y
0.9
Temperature = 0.75
0.3
-1.2
-3
-2
-3
-6.5
47
PATTERN SEARCH
(DIRECT SEARCH)
48
What is a Pattern Search?
An approach that uses a pattern of search directions around
the existing points, the mesh

Polls the mesh for a better solution and moves to that point

Expands/contracts the mesh around the current point when a
solution is not found

Does not rely on gradient information



49
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Pattern Search Overview Iteration 1
Run from specified x0
x
y
3
50
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Pattern Search Overview Iteration 1
Apply pattern vector, poll new points for improvement
x
y
3
Mesh size = 1
Pattern vectors = [1,0], [0,1], [-1,0], [-1,-1]
0
_ * _ x vector pattern size mesh Pnew
0
] 0 , 1 [ * 1 x 1.6
0.4
4.6
2.8
First poll successful
Complete Poll (not default)
51
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Pattern Search Overview Iteration 2
Increase mesh size and repeat
x
y
3
Mesh size = 2
Pattern vectors = [1,0], [0,1], [-1,0], [-1,-1]
1.6
0.4
4.6
2.8
-4
0.3 -2.8
Complete Poll
52
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Pattern Search Overview Iteration 3
Mesh expansion: increase mesh size and repeat
x
y
3
Mesh size = 4
Pattern vectors = [1,0], [0,1], [-1,0], [-1,-1]
1.6
0.4
4.6
2.8
-4
0.3 -2.8
53
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Pattern Search Overview Iteration 4
Refine mesh: decrease mesh size and repeat
x
y
3
Mesh size = 4*0.5 = 2
Pattern vectors = [1,0], [0,1], [-1,0], [-1,-1]
1.6
0.4
4.6
2.8
-4
0.3 -2.8
54
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Pattern Search Overview Iteration N
Continue expansion/contraction until convergence
x
y
3
1.6
0.4
4.6
2.8
-4
0.3 -2.8
-6.5
57
GENETIC ALGORITHM
58
What is a Genetic Algorithm?
Genetic Algorithms use concepts from evolutionary biology to
find exact or approximate solutions to optimization problems

Start with an initial generation of candidate solutions that are
tested against the objective function

Subsequent generations evolve from the 1
st

through selection, crossover and mutation

The individual that best minimizes the given
objective is returned as the ideal solution

59
How Evolution Works Binary Case
Selection
Retain the best performing bit strings from one generation to the
next. Favor these for reproduction
parent1 = [ 1 0 1 0 0 1 1 0 0 0 ]
parent2 = [ 1 0 0 1 0 0 1 0 1 0 ]

Crossover
parent1 = [ 1 0 1 0 0 1 1 0 0 0 ]
parent2 = [ 1 0 0 1 0 0 1 0 1 0 ]
child = [ 1 0 0 0 0 1 1 0 1 0 ]

Mutation
parent = [ 1 0 1 0 0 1 1 0 0 0 ]
child = [ 0 1 0 1 0 1 0 0 0 1 ]



60
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Genetic Algorithm Iteration 1
Evaluate initial population
x
y
61
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Genetic Algorithm Iteration 1
Select a few good solutions for reproduction
x
y
62
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Genetic Algorithm Iteration 2
Generate new population and evaluate
x
y
63
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Genetic Algorithm Iteration 2
Select a few good solutions for reproduction
x
y
64
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Genetic Algorithm Iteration 3
Generate new population and evaluate
x
y
65
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Genetic Algorithm Iteration 3
Select a few good solutions for reproduction
x
y
66
-3 -2 -1 0 1 2 3
-3
-2
-1
0
1
2
3
Genetic Algorithm Iteration N
Continue process until stopping criteria are met
x
y
Solution found
69
Comparison of Solver (Default) Performance
x
y
rf([x,y])


-6 -4 -2 0 2 4 6
-6
-4
-2
0
2
4
6
FuncValue
Min
StartPt
x
y
rf([x,y])


-60 -40 -20 0 20 40 60
-60
-40
-20
0
20
40
60
FuncValue
Min
StartPt
x
y
rf([x,y])


-40 -20 0 20 40 60
-40
-20
0
20
40
60
FuncValue
Min
StartPt
70


71
Additional Resources
Upcoming Webinars
Speeding Up Optimization with Parallel Computing
(August 2010)

On-demand Webinars
Genetic Algorithm in Financial Applications
Tips & Tricks: Getting Started with Optimization
Introduction to Optimization

72
Contact Information
North America
Phone: 508-647-7000
E-mail: support@mathworks.com

Outside North America
Contact your local MathWorks office or reseller:
www.mathworks.com/contact
73
Questions?
74
MATLAB Provides the Foundation for
Optimization
The leading environment for
technical computing

Customizable
Numeric computation
Data analysis and visualization
The de facto industry-standard,
high-level programming language
for algorithm development
Toolboxes for statistics, optimization,
symbolic math, signal and image
processing, and other areas
Foundation of the
MathWorks product family
75
Optimization Toolbox
Solve standard and large-scale optimization problems
Graphical user interface and command
line functions for:
Linear and nonlinear programming
Quadratic programming
Nonlinear least squares and nonlinear
equations
Multi-objective optimization
Binary integer programming

Additional Capabilities:
Parallel computing support in selected
solvers
Customizable algorithm options
Choose between standard and large-
scale algorithms
Output diagnostics
76
Global Optimization Toolbox
Solve multiple maxima, multiple minima, and nonsmooth optimization problems
Graphical user interface and command
line functions for:
Global Search solver
Multistart solver
Genetic algorithm solver
Single objective
Multiobjective with Pareto front
Direct search solver
Simulated annealing solver

Useful for problems not easily addressed
with Optimization Toolbox:
Discontinuous
Highly nonlinear
Stochastic
Discrete or custom data types
Undefined derivatives
Multiple maxima/minima
77
Anatomy of an Optimization Problem
General Form Accepted by MATLAB Solvers
Objective Function



Subject to Constraints (i.e. such that)
u x l
x c b x A
x c b Ax
eq eq eq



0 ) ( ,
0 ) ( ,
) ( min x f
x
Typically a linear or nonlinear function
Linear constraints
inequalities
equalities
bounds
Nonlinear constraints
inequalities
equalities
Decision variables
78
MATLAB Optimization Products and Example
Applications
MATLAB Statistics Toolbox
Curve Fitting Toolbox
Optimization Toolbox Genetic Algorithm and
Direct Search Toolbox
Solving Equations

f(x) = 0
Root finding
Systems of equations

Real roots finding (1D):






Linear Systems:
F(x) = 0
Ax-b = 0 (i.e. Ax=b)
>> x = A \ b
Real roots finding (N-D):







Nonlinear Systems
F(X) = 0
Noisy, discontinuous root
finding (N-D)






Noisy, discontinuous systems:
F(X) = 0
Curve/Modeling
Fitting

curve fitting
parameter estimation
(model fitting)
Basic (linear) curve fitting Advanced (nonlinear) curve
fitting
Model Fitting (least squares)
Constrained curve fitting

Parameter Estimation

Noisy, Discontinuous
parameter estimation
Trade-Off Studies
Maximization
Minimization
Goal seeking
Multiobjective
Unconstrained nonlinear
minimization
Constrained nonlinear
minimization
Noisy, Discontinuous, ill-
defined mimization
-6 -4 -2 0 2 4 6
-100
-50
0
50
100
x
x
3
-2 x-5
0 0.5 1 1.5 2 2.5
0.5
0.6
0.7
0.8
0.9