Design and Analysis of

Algorithm


REVISION

UNIT I
ALGORITHM
ANALYSIS
Wednesday, July 11, 2012 Design and Analysis of Computer Algorithm
2
OUTLINE
Time Space Tradeoff
Asymptotic Notations
Conditional asymptotic notation
Recurrence equations
Analysis of linear search
Wednesday, July 11, 2012 Design and Analysis of Algorithm
3
What is Algorithm?
An algorithm is a sequence of unambiguous
instructions for solving a problem, i.e., for
obtaining a required output for any legitimate
input in a finite amount of time.
An algorithm is a finite set of instructions
that, if followed, accomplishes a
particular task. In addition, all algorithms
must satisfy the following criteria:


Design and Analysis of Algorithm
What is Algorithm?
In addition, all algorithms must satisfy the
following criteria:

1) input
2) Output
3) Definiteness
4) Finiteness
5) Effectiveness
Design and Analysis of Algorithm
Asymptotic notations
Asymptotic Theta notation O
O(g(n)) = {f(n)| there are positive constants c,d
and n
0
such that cg(n) s f(n)s dg(n) for all n>n
0
}
Asymptotic notations
Asymptotic Omega notation O
O(g(n)) = {f(n)| there are positive constants c and n
0

such that 0 s cg(n) s f(n) for all n>n
0
}
Asymptotic notations
Asymptotic Big O notationO
O(g(n)) = {f(n)| there are positive constants c and n
0
such
that 0s f(n)s cg(n) for all n>n
0
}

Algorithm Design Methods
Greedy method.
Divide and conquer.
Dynamic Programming.
Backtracking.
Branch and bound.
Design and Analysis of Algorithm
UNIT II
DIVIDE AND
CONQUER

Outline
Divide and Conquer:
Binary Search
Finding Maximum and Minimum
Merge Sort

Greedy Algorithms:
Container Loading
Knapsack Problem
12
Binary Search
13
Binary Search: middle element
High + low
2
mid =
Steps
Divide the problem into a number of sub
problems.
Conquer the sub problems by solving them
recursively.
If the sub problem sizes are small enough,
however, just solve the sub problems in a
straightforward manner.
Combine the solutions to the sub problems
into the solution for the original problem.
Steps in Merge Sort
Divide
Divide the sequence to be sorted into two
halves.
Conquer
Sort the two subsequences recursively using
merge sort.
Combine
Merge the two sorted subsequences to produce
the sorted answer.
Merge Sort: Example
36 86 45 12 84 77 51 64
36 86 45 12 84 77 51 64
36 86 45 12 84 77 51 64
12 45 77 84 51 64
36 86 12 45 77 84 51 64
36 86 12 45 77 84 51 64
divide
divide divide
divide
36 86
conquer
36 86
combine
divide
45 12
conquer
combine
conquer
combine
84 77 51 64
Optimization Problem
A problem in which some function (called
the optimization or objective function) is
to be optimized (usually minimized or
maximized) subject to some constraints.

Feasible And Optimal Solutions
A feasible solution is a solution that
satisfies the constraints.

An optimal solution is a feasible solution
that optimizes the objective/optimization
function.
Container Loading
Ship has capacity c.
m containers are available for loading.
Weight of container i is w
i
.
Each weight is a positive number.
Sum of container weights > c.
Load as many containers as is possible
without sinking the ship.
Greedy Solution
Load containers in increasing order of
weight until we get to a container that
doesnt fit.
Does this greedy algorithm always load
the maximum number of containers?
Yes. May be proved using a proof by
induction (see text).
0/1 Knapsack Problem
0/1 Knapsack Problem
Hiker wishes to take n items on a trip.
The weight of item i is w
i
.
The items are to be carried in a knapsack
whose weight capacity is c.
When sum of item weights <= c, all n items
can be carried in the knapsack.
When sum of item weights > c, some items
must be left behind.
Which items should be taken/left?
0/1 Knapsack Problem
Hiker assigns a profit/value p
i
to item i.
All weights and profits are positive numbers.
Hiker wants to select a subset of the n items to
take.
The weight of the subset should not exceed the
capacity of the knapsack. (constraint)
Cannot select a fraction of an item. (constraint)
The profit/value of the subset is the sum of the
profits of the selected items. (optimization
function)
The profit/value of the selected subset should
be maximum. (optimization criterion)
0/1 Knapsack Problem
Let x
i
= 1 when item i is selected and let x
i
= 0
when item i is not selected.

i = 1
n
p
i
x
i
maximize
i = 1
n
w
i
x
i
<= c
subject to
and x
i
= 0 or 1 for all i
Jaruloj Chongstitvatana

UNIT III
DYNAMIC
PROGRAMMING


OUTLINE
Multistage Graphs
All-Pair shortest paths
Optimal binary search trees
0/1 Knapsack
Travelling salesperson problem
Design and Analysis of Computer Algorithm
Dynamic programming approach
Dynamic programming approach (forward approach):






d(S, T) = min{1+d(A, T), 2+d(B, T), 5+d(C, T)}
S T
2
B
A
C
1
5
d(C, T)
d(B, T)
d(A, T)
A
T
4
E
D
11
d(E, T)
d(D, T)
d(A,T) = min{4+d(D,T), 11+d(E,T)}
= min{4+18, 11+13} = 22.

S T
13 2
B E
9
A D
4
C F
2
1
5
11
5
16
18
2
7 -28
d(B, T) = min{9+d(D, T), 5+d(E, T), 16+d(F, T)}
= min{9+18, 5+13, 16+2} = 18.






d(C, T) = min{ 2+d(F, T) } = 2+2 = 4
d(S, T) = min{1+d(A, T), 2+d(B, T), 5+d(C, T)}
= min{1+22, 2+18, 5+4} = 9.

B T
5
E
D
F
9
16
d(F, T)
d(E, T)
d(D, T)
S T
13 2
B E
9
A D
4
C F
2
1
5
11
5
16
18
2
7 -29
Backward approach
(forward reasoning)



d(S, A) = 1
d(S, B) = 2
d(S, C) = 5
d(S,D)=min{d(S,A)+d(A,D), d(S,B)+d(B,D)}
= min{ 1+4, 2+9 } = 5
d(S,E)=min{d(S,A)+d(A,E), d(S,B)+d(B,E)}
= min{ 1+11, 2+5 } = 7
d(S,F)=min{d(S,B)+d(B,F), d(S,C)+d(C,F)}
= min{ 2+16, 5+2 } = 7
S T
13 2
B E
9
A D
4
C F
2
1
5
11
5
16
18
2
7 -30
d(S,T) = min{d(S, D)+d(D, T), d(S,E)+
d(E,T), d(S, F)+d(F, T)}
= min{ 5+18, 7+13, 7+2 }
= 9
S T
13 2
B E
9
A D
4
C F
2
1
5
11
5
16
18
2
All pairs shortest path
The problem: find the shortest path between every pair of
vertices of a graph

The graph: may contain negative edges but no negative
cycles

A representation: a weight matrix where
W(i,j)=0 if i=j.
W(i,j)= if there is no edge between i and j.
W(i,j)=weight of edge

Note: we have shown principle of optimality applies to
shortest path problems
The weight matrix and the
graph
1 2 3 4 5
1 0 1 1 5
2 9 0 3 2
3 0 4
4 2 0 3
5 3 0

v
1
v
2
v
3
v
4
v
5
3
2
2
4
1
3
1
9
3
5
Determine Optimal Binary
Search Tree
} min{
1 , ij l i ij
j l i
c c w + +

s s
If T
ij
is an optimal binary search tree for a
i+1
, , a
j
,
and r
ij
=k, then i< k <j. T
ij
has two subtrees L and R.
L contains a
i+1
, , a
k-1
, and R contains a
k+1
, , a
j
.
So the cost c
ij
of T
ij
is
c
ij
= p
k
+ cost(L) + cost(R) + weight(L) + weight(R)
c
ij
= p
k
+ c
i,k-1
+ c
kj
+ w
i,k-1
+ w
kj
= w
ij
+ c
i,k-1
+ c
kj

Since T
ij
is optimal, we have
w
ij
+ c
i,k-1
+ c
kj
=
} { min
1 , 1 , lj l i
j l i
kj k i
c c c c + = +

s <

} min{
1 , ij l i ij
j l i
c c w + +

s s
Example 10.2
Let n=4, (a1, a2, a3, a4) = (do, if return, while). Let (p1, p2, p3,
p4)=(3,3,1,1) and (q0, q1, q2, q3, q4)=(2,3,1,1,1). w
ii
= q
ii
, c
ii
=0,
and r
ii
=0, 0 i 4.
w
01
= p
1
+ w
00
+ w
11
= p
1
+q
1
+w
00
= 8
c
01
= w
01
+ min{c
00
+c
11
} = 8
r
01
= 1
w
12
= p
2
+ w
11
+ w
22
= p
2
+q
2
+w
11
= 7
c
12
= w
12
+ min{c
11
+c
22
} = 7
r
12
= 2
w
23
= p
3
+ w
22
+ w
33
= p
3
+q
3
+w
22
= 3
c
23
= w
23
+ min{c
22
+c
33
} = 3
r
23
= 3
w
34
= p
4
+ w
33
+ w
44
= p
4
+q
4
+w
33
= 3
c
34
= w
34
+ min{c
33
+c
44
} = 3
r
34
= 4
Example 10.2 Computation
w
00
=2
c
00
=0
r
00
=0
w
11
=3
c
11
=0
r
11
=0
w
22
=1
c
22
=0
r
22
=0
w
00
=2
c
00
=0
r
00
=0
w
33
=1
c
33
=0
r
33
=0
w
44
=1
c
44
=0
r
44
=0
w
01
=8
c
01
=8
r
01
=1
w
12
=7
c
12
=7
r
12
=2
w
23
=3
c
23
=3
r
23
=3
w
34
=3
c
34
=3
r
34
=4
w
02
=12
c
02
=19
r
02
=1
w
13
=9
c
13
=12
r
13
=2
w
24
=5
c
24
=8
r
24
=3
w
03
=14
c
03
=25
r
03
=2
w
14
=11
c
14
=19
r
14
=2
w
04
=16
c
04
=32
r
04
=2
0
1
2
3
4
4
0
1
2
3

UNIT IV
BACKTRACKING

OUTLINE
8 Queens problem
sum of subsets
graph coloring
Hamiltonian problem
knapsack problem
Wednesday, July 11, 2012
37
Design and Analysis of Computer Algorithm
Backtracking 38
Backtracking
Two versions of backtracking algorithms
Solution needs only to be feasible (satisfy
problems constraints)
sum of subsets
Solution needs also to be optimal
knapsack
The backtracking method
A given problem has a set of constraints and possibly an
objective function

The solution optimizes an objective function, and/or is feasible.

We can represent the solution space for the problem using a
state space tree

The root of the tree represents 0 choices,
Nodes at depth 1 represent first choice
Nodes at depth 2 represent the second choice, etc.
In this tree a path from a root to a leaf represents a candidate
solution
Sum of subsets
Problem: Given n positive integers w
1,
...
w
n
and a positive integer S. Find all
subsets of w
1,
... w
n
that sum to S.
Example:
n=3, S=6, and w
1
=2, w
2
=4, w
3
=6

Solutions:
{2,4} and {6}
Backtracking 40
Sum of subsets
We will assume a binary state space tree.

The nodes at depth 1 are for including
(yes, no) item 1, the nodes at depth 2 are
for item 2, etc.

The left branch includes w
i
, and the right
branch excludes w
i
.
The nodes contain the sum of the
weights included so far
Backtracking 41


Sum of subset Problem:


State SpaceTree for 3 items
w
1
= 2, w
2
= 4, w
3
= 6 and S = 6
Backtracking 42
i
1

i
2

i
3

yes
no
0
0
0
0
2
2
2
6
6
12
8
4
4
10
6
yes
yes
no
no
no
no no
no
The sum of the included integers is stored at the node.
yes
yes
yes yes
The Eight Queens Problem
Figure 6-2
A solution to the Eight Queens
problem
Graph Colouring
Goal: Create minimum number groupings of
independent vertices
Independent vertices not connected by an
edge
Examples:
Task scheduling
Register allocation
Frequency allocation
44
Graph Colouring Problem Def
n
Let G=(V,E) be an undirected graph
A k-coloring of G is a function
c : V {1,2,...,k} such that
if c(u) =c(v) then (u,v) e E
45
Example
46
1
2
3
5
4
1
2
3 3
4 4 4
5 5 5
Example
47
1
2
3
1
2
3 3
4
5 5
4
3
Example
48
1
2
3
1
2
3 3
4
5 5
4
3
4 4
5
5 5
Example
49
1
2
3
1
2
3 3
4
5 5
4
3
4 4 4
Example
50
1
2
3
1
2
4
5 5
4
3
2
Example
51
1
2
3
1
2
4
5 5
4
2
3
4 4
4
5 5

UNIT V
BRANCH AND
BOUND
Backtracking 53
Example
Suppose n = 4, W = 16, and we have the
following:

i p
i
w
i
p
i
/ w
i


1 $40 2 $20
2 $30 5 $6
3 $50 10 $5
4 $10 5 $2
Backtracking 54
maxprofit = $0 (n = 4, C = 16 )
Node 1
a) profit = $ 0
weight = 0

b) bound = profit + p
1
+ p
2
+ (C - 7 ) * p
3
/ w
3

= $0 + $40 + $30 + (16 -7) X $50/10 =$115

c) 1 is promising because its weight =0 < C =
16
and its bound $115 > 0 the value of
maxprofit.


The calculation for node 1
Backtracking 55
Item 1 with profit $40 and weight 2 is included
maxprofit = $40
a) profit = $40
weight = 2

b) bound = profit + p
2
+ (C - 7) X p
3
/ w
3

= $40 + $30 + (16 -7) X $50/10
=$115

c) 2 is promising because its weight =2 < C =
16
and its bound $115 > $40 the value of
maxprofit.


The calculation for node 2
Backtracking 56
Item 1 with profit $40 and weight 2 is not included
At this point maxprofit=$90 and is not changed
a) profit = $0
weight = 0

b) bound = profit + p
2
+ p
3
+ (C - 15) X p
4
/ w
4

= $0 + $30 +$50+ (16 -15) X $10/5 =$82

c) 13 is nonpromising because its bound $82 < $90
the value of maxprofit.


The calculation for node 13
Backtracking 57
1
13
$0
0
$115
$40
2
$115
$0
0
$82
Item 1 [$40, 2]
B
82<90
Item 2 [$30, 5]
$70
7
$115
$120
17

2
3
4
F
17>16
Item 3 [$50, 10]
Item 4 [$10, 5]
$40
2
$98
8
$70
7
$80
5
$90
12
$98
9
$40
2
$50
12
B
50<90
$80
12
$80
6
$70
7
$70
7
N
N
$100
17

10
$90
12
$90
11
maxprofit = 90
profit
weight
bound
Example
F - not feasible
N - not optimal
B- cannot lead
to best solution
Optimal
maxprofit =0
maxprofit =40
maxprofit =70
maxprofit =80
F
17>16
NP and NP-Completeness
Bryan Pearsaul
Outline
Decision and Optimization Problems
P and NP
Polynomial-Time Reducibility
NP-Hardness and NP-Completeness
Examples: TSP, Circuit-SAT, Knapsack
Polynomial-Time Approximation Schemes
Decision and Optimization
Problems
Decision Problem: computational problem
with intended output of yes or no, 1 or 0
Optimization Problem: computational
problem where we try to maximize or
minimize some value.
Complexity Class P
Deterministic in nature
Solved by conventional computers in
polynomial time
O(1) Constant
O(log n) Sub-linear
O(n) Linear
O(n log n) Nearly Linear
O(n
2
) Quadratic
Polynomial upper and lower bounds
Complexity Class NP
Non-deterministic part as well
choose(b): choose a bit in a non-deterministic
way and assign to b
If someone tells us the solution to a problem, we
can verify it in polynomial time
Two Properties: non-deterministic method to
generate possible solutions, deterministic
method to verify in polynomial time that the
solution is correct.