Global Optimization with MATLAB

Products

Account Manager

Application Engineer

2011 The MathWorks, Inc.1

Agenda

Introduction to Global Optimization Peaks

Survey of Solvers with Examples 8

6
MultiStart
4
Global Search 2

Pattern Search 0

Simulated Annealing -2

-4
Genetic Algorithm / Multiobjective
-6
Genetic Algorithm 3 2
2
1 0
0
-1 -2
-2
-3
y x
Additional Resources
Local minima

Question & Answer Global minima

2
Optimization Finding answers to problems automatically

Modify Design
Parameters
NO

Initial YES
Model or Objectives Optimal
Design
Prototype Achieved? Design
Parameters

OPTIMIZATION PROCESS

Design process can be performed: Optimization benefits include:

Finding better (optimal) designs
Manually Automatically Faster design evaluations
(trial-and-error or iteratively) (using optimization techniques) Useful for trade-off analysis (N dimensions)
Non-intuitive designs may be found

Antenna Design Using Genetic Algorithm 3

http://ic.arc.nasa.gov/projects/esg/research/antenna.htm
Example Global Optimization Problems
Why does fmincon have a hard time finding the
function minimum?
Starting at 0 Starting at 1 Starting at 3
Why didnt fminunc find the maximum volumetric
10 10 10
efficiency?
x sin(x) + x cos(2 x)

5 5 5

0 0 0
Peak VE Value = 0.96144
-5 -5 -5 1
-10 -10 -10 0.9
0 5 10 0 5 10 0 5 10 0.9
x x x
Starting at 6 Starting at 8 Starting at 10 0.8
0.8
10 10 10
x sin(x) + x cos(2 x)

5 5 5 0.7 0.7

Manifold Pressure Ratio

0 0 0
0.6 End 0.6
-5 -5 -5

-10 -10 -10

0.5 0.5
0 5 10 0 5 10 0 5 10
x x x
0.4
Why didnt nonlinear regression find a good fit? 0.4

c=b1e-b4t+b2e-b5t+b3e-b6t 0.3
0.8
Start 0.3
0.2
0.7
0.2
0.6 0.1
0.1
0.5
0
1000 2000 3000 4000 5000 6000
0.4
c

Revolutions Per Minute, RPM

0.3

0.2

0.1

0
0 20 40 60
t
80 100 120
4
Global Optimization

Goal:
Want to find the lowest/largest value of
the nonlinear function that has many local
minima/maxima

Problem:
Traditional solvers often return one of the
local minima (not the global)

Solution:
A solver that locates globally optimal
Rastrigins Function
solutions

5
Global Optimization Toolbox For problems that
contain multiple maxima/minima or are non-smooth

Optimization T Global Optimization

oolbox Toolbox
Faster/fewer function eval
uations
Larger problems (higher d
imensions)
Finds local minima/maxim
a
Finds global minima/maxi
ma (most of the time)
Better on
non-smooth
stochastic

discontinuous
undefined gradients

Custom data types
(in GA and SA solvers)

6
MULTISTART

7
What is MultiStart?

Run a local solver from

each set of start points

Option to filter starting

points based feasibility

Supports parallel
computing

8
GLOBAL SEARCH

11
What is GlobalSearch?

Multistart heuristic algorithm

Calls fmincon from multiple

start points to try and find a
global minimum

Filters/removes non-promising
start points

12
GlobalSearch Overview
Schematic Problem

Peaks function
2 Three minima
Green, z = -0.065
Red, z= -3.05
1
Blue, z = -6.55

0
y

-1

-2

-3
-3 -2 -1 0 1 2 3

x
13
GlobalSearch Overview Stage 0
Run from specified x0

0
y

-1

-2

-3
-3 -2 -1 0 1 2 3

x
14
GlobalSearch Overview Stage 1
Generate stage 1 start points via Scatter Search

2 6
4

1
3

0
y 0
0

-1
-2

-2

-3 0 0
-3 -2 -1 0 1 2 3

x
15
GlobalSearch Overview Stage 1
Find stage 1 start point with lowest penalty value

2 6
4

1
3

0
y 0
0

-1
-2

-2

-3 0 0
-3 -2 -1 0 1 2 3

x
16
GlobalSearch Overview Stage 1
Run from best stage 1 point

0
y

-1

-2

-3
-3 -2 -1 0 1 2 3

x
17
GlobalSearch Overview Stage 2
Generate stage 2 start points using Scatter Search

0
y

-1

-2

-3
-3 -2 -1 0 1 2 3

x
18
GlobalSearch Overview Stage 2
Analyse each stage 2 point in turn.

0
y

-1

-2

-3
-3 -2 -1 0 1 2 3

x
19
GlobalSearch Overview Stage 2
Dont run points that are in basins of existing
minimum
3

0
y

-1

-2

-3
-3 -2 -1 0 1 2 3

x
20
GlobalSearch Overview Stage 2
Analyse each stage 2 point in turn.

0
y

-1

-2

-3
-3 -2 -1 0 1 2 3

x
21
GlobalSearch Overview Stage 2
Dont run points whose penalty value exceeds
threshold
3

Current penalty
2
6 threshold value : 4

0
y

-1

-2

-3
-3 -2 -1 0 1 2 3

x
22
GlobalSearch Overview Stage 2
Analyse each stage 2 point in turn

0
y

-1

-2

-3
-3 -2 -1 0 1 2 3

x
23
GlobalSearch Overview Stage 2
Run start point if it satisfies distance & merit criteria

Current penalty
2
threshold value : 4

0
y

-1

-2
-3

-3
-3 -2 -1 0 1 2 3

x
24
GlobalSearch Overview Stage 2
Expand basin of attraction if minimum already found

Current penalty threshold value : 2

0
y Basins can overlap

-1

-2

-0.1
-3
-3 -2 -1 0 1 2 3

x
25
SIMULATED ANNEALING

28
What is Simulated Annealing?

A probabilistic metaheuristic approach based upon the

physical process of annealing in metallurgy.

Controlled cooling of a metal allows atoms to realign from a

random higher energy state to an ordered crystalline
(globally) lower energy state

By analogy, simulated annealing replaces a current solution

by randomly choosing a nearby solution

A nearby solution is determined by the solution

temperature

29
Simulated Annealing Overview Iteration 1
Run from specified x0

0
y 0.9

-1

-2

-3
-3 -2 -1 0 1 2 3

x
30
Simulated Annealing Overview Iteration 1
Randomly generate a new point according to probability distribution and
current temperature
3

Temperature = 1
2

1
3

0
y 0.9

Possible New Points:

-1
Standard Normal N(0,1) * Temperature

-2

-3
-3 -2 -1 0 1 2 3

x
31
Simulated Annealing Overview Iteration 1
If lower, accept the point, if higher, accept based upon acceptance
probability

Temperature = 1
2

1
Paccept ( 30.9 ) / T
0.11
1 e
1
3

0
y 0.9

-1

-2

-3
-3 -2 -1 0 1 2 3

x
32
Simulated Annealing Overview Iteration 1
Randomly generate a new point according to probability distribution and
current temperature
3

Temperature = 1
2

1
3

0
y 0.9

-1
0.3

-2

-3
-3 -2 -1 0 1 2 3

x
33
Simulated Annealing Overview Iteration 1
Randomly generate a new point according to probability distribution and
current temperature, accept new point if lower value
3

Temperature = 1
2

1
3

0
y 0.9

-1
0.3

-2

-3
-3 -2 -1 0 1 2 3

x
34
Simulated Annealing Overview Iteration 2
Lower temperature according to temperature schedule

Temperature = 1
2

1
3

0
y 0.9

-1
0.3

-2

-3
-3 -2 -1 0 1 2 3

x
35
Simulated Annealing Overview Iteration 2
Lower temperature according to temperature schedule and generate new
point
3

Temperature = 0.75
2

1
3

0
y 0.9
-1.2
-1
0.3

-2

-3
-3 -2 -1 0 1 2 3

x
36
Simulated Annealing Overview Iteration N-1
After several iterations, the search radius becomes small and we narrow
in on a local solution
3

Temperature = 0.1
2

1
3
-3
0
y 0.9
-1.2
-1
0.3

-2

-3
-3 -2 -1 0 1 2 3

x
37
Simulated Annealing Overview Iteration N
Reset temperature and start the process again (reanneling)

Temperature = 1
2

1
3
-3

y 0 -2
0.9
-1.2
-1
0.3

-2

-3
-3 -2 -1 0 1 2 3

x
38
Simulated Annealing Overview Iteration N
Reset temperature and start the process again (reannealing)

Temperature = 1
2

1
3
-3

y 0 -2
1
0.9
-1.2 Paccept 0.27
1 e( 2( 3)) / T
-1
0.3

-2

-3
-3 -2 -1 0 1 2 3

x
39
Simulated Annealing Overview Iteration N
Reset temperature and start the process again (reannealing)

Temperature = 1
2

1
3
-3

y 0 -2
0.9
-1.2
-1
0.3

-2

-3
-3 -2 -1 0 1 2 3

x
40
Simulated Annealing Overview Iteration N+1
Reduce temperature and continue

Temperature = 0.75
2

1
3
-3

y 0 -2
0.9
-1.2
-1
0.3 -3

-2

-3
-3 -2 -1 0 1 2 3

x
41
Simulated Annealing Overview Iteration N+1
Reduce temperature and continue

Temperature = 0.75
2

1
3
-3

y 0 -2
0.9
-1.2
-1
0.3 -3

-2

-3
-3 -2 -1 0 1 2 3

x
42
Simulated Annealing Overview Iteration
Reduce temperature and continue

Temperature = 0.75
2

1
3
-3

y 0 -2
0.9
-1.2
-1
0.3 -3

-2 -6.5

-3
-3 -2 -1 0 1 2 3

x
43
PATTERN SEARCH
(DIRECT SEARCH)

47
What is a Pattern Search?

An approach that uses a pattern of search directions around

the existing points, the mesh

Polls the mesh for a better solution and moves to that point

Expands/contracts the mesh around the current point when a

solution is not found

Does not rely on gradient information

48
Pattern Search Overview Iteration 1
Run from specified x0

1
3

0
y

-1

-2

-3
-3 -2 -1 0 1 2 3

x
49
Pattern Search Overview Iteration 1
Apply pattern vector, poll new points for improvement

3
Mesh size = 1
Pattern vectors = [1,0], [0,1], [-1,0], [-1,-1]
4.6
2
Pnew mesh _ size * pattern _ vector x0
First poll successful
1
3
2.8 1.6 1*[1,0] x0

0
y 0.4

-1 Complete Poll (not default)

-2

-3
-3 -2 -1 0 1 2 3

x
50
Pattern Search Overview Iteration 2
Increase mesh size and repeat

3
Mesh size = 2
Pattern vectors = [1,0], [0,1], [-1,0], [-1,-1]
4.6
2

1
2.8 3
1.6

0
y 0.4
-2.8 0.3

-1

-2
-4
Complete Poll
-3
-3 -2 -1 0 1 2 3

x
51
Pattern Search Overview Iteration 3
Mesh expansion: increase mesh size and repeat

3
Mesh size = 4
Pattern vectors = [1,0], [0,1], [-1,0], [-1,-1]
4.6
2

1
2.8 3
1.6

0
y 0.4
-2.8 0.3

-1

-2
-4

-3
-3 -2 -1 0 1 2 3

x
52
Pattern Search Overview Iteration 4
Refine mesh: decrease mesh size and repeat

3
Mesh size = 4*0.5 = 2
Pattern vectors = [1,0], [0,1], [-1,0], [-1,-1]
4.6
2

1
2.8 3
1.6

0
y 0.4
-2.8 0.3

-1

-2
-4

-3
-3 -2 -1 0 1 2 3

x
53
Pattern Search Overview Iteration N
Continue expansion/contraction until convergence

4.6
2

1
2.8 3
1.6

0
y 0.4
-2.8 0.3

-1
-6.5

-2
-4

-3
-3 -2 -1 0 1 2 3

x
54
GENETIC ALGORITHM

57
What is a Genetic Algorithm?

Genetic Algorithms use concepts from evolutionary biology to

find exact or approximate solutions to optimization problems

Start with an initial generation of candidate solutions that are

tested against the objective function

Subsequent generations evolve from the 1st

through selection, crossover and mutation

The individual that best minimizes the given

objective is returned as the ideal solution

58
How Evolution Works Binary Case

Selection
Retain the best performing bit strings from one generation to the
next. Favor these for reproduction
parent1 = [ 1 0 1 0 0 1 1 0 0 0 ]
parent2 = [1 0 0 1 0 0 1 0 1 0]

Crossover
parent1 = [1 0 1 0 0 1 1 0 0 0]
parent2 = [1 0 0 1 0 0 1 0 1 0]
child = [1 0 0 0 0 1 1 0 1 0]

Mutation
parent = [1 0 1 0 0 1 1 0 0 0]
child = [0 1 0 1 0 1 0 0 0 1]
59
Genetic Algorithm Iteration 1
Evaluate initial population

0
y

-1

-2

-3
-3 -2 -1 0 1 2 3

x
60
Genetic Algorithm Iteration 1
Select a few good solutions for reproduction

0
y

-1

-2

-3
-3 -2 -1 0 1 2 3

x
61
Genetic Algorithm Iteration 2
Generate new population and evaluate

0
y

-1

-2

-3
-3 -2 -1 0 1 2 3

x
62
Genetic Algorithm Iteration 2
Select a few good solutions for reproduction

0
y

-1

-2

-3
-3 -2 -1 0 1 2 3

x
63
Genetic Algorithm Iteration 3
Generate new population and evaluate

0
y

-1

-2

-3
-3 -2 -1 0 1 2 3

x
64
Genetic Algorithm Iteration 3
Select a few good solutions for reproduction

0
y

-1

-2

-3
-3 -2 -1 0 1 2 3

x
65
Genetic Algorithm Iteration N
Continue process until stopping criteria are met

0
y

-1 Solution found

-2

-3
-3 -2 -1 0 1 2 3

x
66
 6
Comparison of Solver (Default) Performance
rf([x,y])

FuncValue
Min
4 StartPt

0
y

-2

-4

-6
-6 -4 -2 0 2 4 6
x
rf([x,y])
60
FuncValue
Min
40 StartPt

20

0
y

-20

-40

-60
-60 -40 -20 0 20 40 60
x
rf([x,y])

FuncValue
60
Min
StartPt

40

20
y

-20

-40

-40 -20 0
x
20 40 60 69
70
Additional Resources

Upcoming Webinars
Speeding Up Optimization with Parallel Computing
(August 2010)

On-demand Webinars
Genetic Algorithm in Financial Applications
Tips & Tricks: Getting Started with Optimization
Introduction to Optimization

71
Contact Information

North America
Phone: 508-647-7000
E-mail: support@mathworks.com

Outside North America

Contact your local MathWorks office or reseller:
www.mathworks.com/contact

72
Questions?

73
MATLAB Provides the Foundation for
Optimization

The leading environment for

technical computing

Customizable
Numeric computation
Data analysis and visualization
The de facto industry-standard,
high-level programming language
for algorithm development
Toolboxes for statistics, optimization,
symbolic math, signal and image
processing, and other areas
Foundation of the
MathWorks product family

74
Optimization Toolbox
Solve standard and large-scale optimization problems

Graphical user interface and command

line functions for:
Linear and nonlinear programming
Quadratic programming
Nonlinear least squares and nonlinear
equations
Multi-objective optimization
Binary integer programming

Additional Capabilities:
Parallel computing support in selected
solvers
Customizable algorithm options
Choose between standard and large-
scale algorithms
Output diagnostics

75
Global Optimization Toolbox
Solve multiple maxima, multiple minima, and nonsmooth optimization problems

Graphical user interface and command

line functions for:
Global Search solver
Multistart solver
Genetic algorithm solver
Single objective
Multiobjective with Pareto front
Direct search solver
Simulated annealing solver

Useful for problems not easily addressed

with Optimization Toolbox:
Discontinuous
Highly nonlinear
Stochastic
Discrete or custom data types
Undefined derivatives
Multiple maxima/minima

76
Anatomy of an Optimization Problem

General Form Accepted by MATLAB Solvers

Objective Function
Typically a linear or nonlinear function
min f ( x)
x

Decision variables

Subject to Constraints (i.e. such that)

Ax b, c( x) 0
Aeq x beq , ceq ( x) 0
Nonlinear constraints
Linear constraints l xu inequalities
inequalities equalities
equalities
bounds

77
 MATLAB Optimization Products and Example
Applications
MATLAB Statistics Toolbox Optimization Toolbox Genetic Algorithm and
Curve Fitting Toolbox Direct Search Toolbox

Solving Equations Real roots finding (1D): Real roots finding (N-D): Noisy, discontinuous root
100
x 3-2 x-5 finding (N-D)
50

f(x) = 0 0

Root finding -50

Systems of equations -100

-6 -4 -2 0 2 4 6
x

Linear Systems:
F(x) = 0 Nonlinear Systems Noisy, discontinuous systems:
Ax-b = 0 (i.e. Ax=b) F(X) = 0 F(X) = 0
>> x = A \ b

Curve/Modeling Basic (linear) curve fitting Advanced (nonlinear) curve Model Fitting (least squares) Noisy, Discontinuous
fitting Constrained curve fitting parameter estimation
Fitting 0.9

0.8
Parameter Estimation
0.7
curve fitting
0.6
parameter estimation
0.5
(model fitting) 0 0.5 1 1.5 2 2.5

Trade-Off Studies Unconstrained nonlinear Constrained nonlinear Noisy, Discontinuous, ill-

minimization minimization defined mimization
Maximization
Minimization
Goal seeking
Multiobjective

78