01 Iterative Methods

Iterative Algorithms
& Loop Invariants
Lecture 1
Understanding Algorithms Abstractly

Home to School Problem
Steps in an Iterative Algorithm
Assertions
Insertion Sort
Designing an Algorithm
Finding Errors
Typical Loop Invariants
Binary Search Like Examples
The Partition Problem
Greatest Common Divisor
Bucket (Quick) Sort for Humans
Jeff Edmonds Radix/Counting Sort
York University Lower Bound for Sort COSC 31011
The Goal is to
UNDERSTAND
Algorithms
Fundamentally to their core
Representation:
Understand the relationship between
different representations of the same
information or idea
1
2
3
Rudich www.discretemath.com
Different Representations
of Algorithms
Code
Running example
DFA and Turing Machines
Higher level abstract view
Code
Representation of an Algorithm
class InsertionSortAlgorithm extends SortAlgorithm {
void sort(int a[]) throws Exception {
for (int i = 1; i < a.length; i++) {
int j = i;
int B = a[i];
while ((j > 0) && (a[j-1] > B)) {
a[j] = a[j-1];
j--; }
a[j] = B;
}}
Pros and Cons?

5
Code
Pros:
Cons:
Runs on computers
Precise and succinct
Perception that being
able to code is the
only thing needed to
get a job. (false)
I am not a computer
I need a higher level of
intuition.
Prone to bugs
Language dependent
Running Example
Try out a problem or
solution on small examples.
Running Example
52
88
14
31
25
98
30
88
23
62
14
98
79
Running Example
14,23,25,30,31,52,62,79,88,98
Pros and Cons?

9
Running Example
Pros:
Concrete
Dynamic
Visual
Cons:
Relies on you to find the
pattern.
Does not explain why it
works.
Demonstrates for only
one of many inputs.
10

Pros and Cons?

11

Pros:
Good for theorems
about algorithms
Cons:
Not good for designing
algorithms
12
Higher Level Abstract View

1
T+
i-1
13
Levels of Abstraction
It is hard to think of love in terms of the firing of
neurons.
vs
Software developers view subsystems as entities with

separate personalities, roles, and interactions,
not details of code.
vs
14

Pros:
Intuitive for humans
Useful for
thinking about
designing
describing algorithms
Cons:
Mathematical mumbo
jumbo
Too abstract
Students resist it
View from which

correctness is self
evident.
15
Value Simplicity
Abstract away the inessential features of a problem
=
Goal: Understand and think about complex
algorithm in simple ways.
Dont tune out. There are deep ideas within
the simplicity.
16
Given a concrete algorithm, eg. MergeSort
and a concrete input,
eg. <4,2,6,1,7>
trace it and prove that it works.
Given a concrete algorithm, eg. MergeSort
and an abstract input,
eg. <x1, xn>
Given a Meta Algorithm, eg. Iterative Algs
and an abstract input,
eg. <x1, xn>
17
Given a new computational problem on exam,
design a new concrete algorithm that solves it
following the Meta Algorithm steps.
If you do not get all the details of the new
algorithm correct, it wont be a disaster.
If you do not get the steps of the Meta
Algorithm correct, it will be a disaster.
There are only a few Meta Algorithms
Learn them!
18
Meta Algorithms
Recursive Algorithms
Greedy Algorithms
Recursive Back Tracking
Dynamic Programming.
NP-Completeness
19
Meta Math
Existential and Universal Quantifiers
Asymptotic Growth
Adding Made Easy Approximations
Recurrence Relations
20
Value Correctness
Dont tack on a formal proof of correctness
after coding to make the professor happy.
It need not be mathematical mumbo jumbo.
Goal: To think about algorithms in such way
that their correctness is transparent.
21
Explaining an Algorithm
Goal: To provide a number of
different notations, analogies, and
abstractions within which to
develop, think about, and
describe algorithms.
22
Two Key Types of Algorithms

23
loop (until done)
take step
end loop
A good way to structure

many programs:
Store the key information
you currently know in some
data structure.
In the main loop, take a step
forward towards destination
by making a simple change
to this data.
24
I like understanding things

using a story.
I will now tell one.
25
The Getting to School Problem
26
Problem Specification
Pre condition: location of home and school
Post condition: traveled from home to school
27
Algorithm
What is an Algorithm?
28
Algorithm
The algorithm defines the computation route.
29
Algorithm
Is this reasonable?
30
Complexity
There are an infinite number of input instances
Algorithm must work for each.
31
Complexity
Difficult to predict where computation might be
in the middle of the computation.
32
Location of Computation
The current State of computation is
determined by values of all variables.
33
Suppose the computation ends up here?
34
Dont Panic
Where ever the computation might be,
take best step towards school.
35
General Principle
Do not worry about the entire computation.
Take one step at a time!
36
Defining Algorithm
Wherever the computation might be,
37
Take a step
What is required of this step?
38
A Measure of Progress
75 km
to school
79 km
to school
39
Defining Algorithm
Is this sufficient to define a working algorithm?
79 km
75 km
40
Working Algorithm
Computation steadily approaches goal
79 km
75 km
to school
to school
41
Defining Algorithm
Extra requirements
79 km
75 km
79 km
km
0 km
78.999 km
42
Define a Step
Wherever the computation might be,
43
Computation Stuck
Some location too confusing for algorithm
Algorithm too lazy to define step for every
location.
44
Safe Locations
Algorithm specifies from which locations
it knows how to step.
45
Loop Invariant
The computation is presently in a safe location.
Maybe true and maybe not.
If not something has gone wrong.
46
Defining Algorithm
From every safe location,
define one step towards school.
47
Take a step
What is required of this step?
48
Maintain Loop Invariant

If the computation is in a safe location,
it does not step into an unsafe one.
49

Can we be assured that the computation
will always be in a safe location?
50

Can we be assured that the computation
will always be in a safe location?
No. What if it is not initially true?

51
Establishing Loop Invariant

From the Pre-Conditions on the input instance
we must establish the loop invariant.
52
Can we be assured that the

computation will always be
in a safe location?
By what principle?
53

By Induction the computation will
always be in a safe location.
S (0 )
iS ( i)
i S ( i ) S ( i 1 )
54
Ending The Algorithm

Define Exit Condition
Exit
Termination:
With sufficient progress,
the exit condition will be met.
When we exit, we know
exit condition is true
loop invariant is true
from these we must establish
the post conditions.
0 km
Exit
Exit
Exit
55
Lets Recap
56
Designing an Algorithm Is this sufficient?

Define Problem
Define Loop
Invariants
Define Measure of
Progress
79 km
to school
Define Step
Define Exit Condition Maintain Loop Inv

Exit
Exit
Make Progress
Initial Conditions
Ending
Exit
79 km
75 km
km
0 km
Exit
Exit
57
Consider an Algorithm
Exit
Exit
79 km
75 km
km
0 km
Exit
Exit
58
Loop Invariant Maintained
Exit
59
Computation can always proceed
60
Computation always makes progress
79 km
75 km
Exit
79 km
75 km
61
Computation Terminates
km
0 km
Exit
0 km
79 km
75 km
0 km
Exit
62
Computation Terminates
Exit
Exit
63
Consider an Algorithm
Exit
Exit
79 km
75 km
km
0 km
Exit
Exit
This is sufficient!
64
This completes
One Higher Level Abstract View
of Iterative Algorithms
65
We will now redo

the previous ideas
a little more formally
Assertions
66
Paradigm Shift
Is the black the form and the green the background?

Is the green the form and the black the background?
It is help to have different ways of looking at it.
67
A Sequence of Actions
Max( a,b,c )
m=a
if( b>m )
m=b
endif
if( c>m )
m=c
endif
return(m)
If you are describing an alg to a

friend, is this what you say?
Do you think about an algorithm as
a sequence of actions?
Do you explain it by saying:
Do this. Do that. Do this?
Do you get lost about
where the computation is
and where it is going?
What if there are many paths
through many ifs and loops?
How do you know it works
for every path
on every input?
68
Max( a,b,c )
preCond: Input has 3 numbers.
m=a
if( b>m )
m=b
endif
if( c>m )
m=c
endif
return(m)
postCond:
return max in {a,b,c}
At least tell me what the algorithm

is supposed to do.
Preconditions:
Any assumptions that must be true
about the input instance.
Postconditions:
The statement of what must be true
when the algorithm/program
returns.
69
Max( a,b,c )
m=a
if( b>m )
m=b
endif
How can you possibly understand

this algorithm without knowing
what is true when the computation
is here?
if( c>m )
m=c
endif
return(m)
postCond:
70
Max( a,b,c )
m=a
if( b>m )
m=b
endif
assert: m is max in {a,b}
if( c>m )
m=c
endif
How can you possibly understand

this algorithm without knowing
what is true when the computation
is here?
Tell me!
return(m)
postCond:
71
Value Correctness
Dont tack on a formal proof of correctness
after coding to make the professor happy.
It need not be mathematical mumbo jumbo.
Goal: To think about algorithms in such way
that their correctness is transparent.
72
A Sequencevs of Actions
A Sequence of Assertions
Max( a,b,c )
m=a
assert: m is max in {a}
if( b>m )
m=b
endif
if( c>m )
m=c
endif
assert: m is max in {a,b,c}
return(m)
postCond:
It is help to have
different ways of
looking at it.
73
Purpose of Assertions
Useful for
thinking about algorithms
developing
describing
proving correctness
74
Definition of Assertions
An assertion is a statement
about the current state of the data
structure that is either true or false.
eg. the amount in your bank account
is not negative.
75
It is made at some particular point during
the execution of an algorithm.
It should be true independent of
the path followed through the code
and independent of the input.
If it is false, then something has gone
wrong in the logic of the algorithm.
76
An assertion is not a task for the
algorithm to perform.
It is only a comment that is added
for the benefit of the reader.
77
Don't Be Frightened
An assertion need not consist of formal
mathematical mumble jumble
Use an informal description
and a picture
78
Specifying a Computational
Problem
Max( A )
preCond:
Input is array A[1,n]
of n values.
i=1
m = A[1]
loop
exit when (i=n)
i=i+1
m = max(m,A[i])
endloop
return(m)
postCond:
m is max in A[1,n]
Preconditions:
Any assumptions that must be
true about the input instance.
Postconditions:
The statement of what must
be true when the
algorithm/program returns.
79
Definition of Correctness
<PreCond> & <code> <PostCond>
If the input meets the preconditions,
then the output must meet the
postconditions.
If the input does not meet the preconditions,
then nothing is required.
80
<assertion0>
if( <condition1> ) then
code<1,true>
else
code<1,false>
end if
<assertion1>
<assertionr-1>
if( <conditionr> ) then
code<r,true>
else
code<r,false>
end if
<assertionr>
<assertion0>
any <conditions>
code
<assertionr>
How is this proved?
81
<assertion0>
code<1,true>
else
code<1,false>
end if
<assertion1>
<assertionr-1>
code<r,true>
else
code<r,false>
end if
<assertionr>
<assertion0>
any <conditions>
code
<assertionr>
Must check 2r different

settings of <conditions>
paths through the code.
Is there a faster way?
82
<assertion0>
code<1,true>
else
code<1,false>
end if
<assertion1>
<assertionr-1>
code<r,true>
else
code<r,false>
end if
<assertionr>
Step 1
<assertion0>
<condition1>
code<1,true>
<assertion1>
<assertion0>
<condition1>
code<1,false>
<assertion1>
83
<assertion0>
code<1,true>
else
code<1,false>
end if
<assertion1>
<assertionr-1>
code<r,true>
else
code<r,false>
end if
<assertionr>
Step 2
<assertion1>
<condition2>
code<2,true>
<assertion2>
<assertion1>
<condition2>
code<2,false>
<assertion2>
84
<assertion0>
code<1,true>
else
code<1,false>
end if
<assertion1>
<assertionr-1>
code<r,true>
else
code<r,false>
end if
<assertionr>
Step r
<assertionr-1>
<conditionr>
code<r,true>
<assertionr>
<assertionr-1>
<conditionr>
code<r,false>
<assertionr>
85
Max( a,b,c )
preCond:
Input has 3 numbers.
m=a
if( b>m )
m=b
endif
if( c>m )
m=c
endif
return(m)
postCond:
Example
<preCond>
code
<assertion0>
86
Max( a,b,c )
preCond:
m=a
if( b>m )
m=b
endif
if( c>m )
m=c
endif
return(m)
postCond:
Example
<assertion0>
<condition1>
code<1,true>
<assertion1>
<assertion0>
<condition1>
code<1,false>
<assertion1>
87
Max( a,b,c )
preCond:
m=a
if( b>m )
m=b
endif
if( c>m )
m=c
endif
return(m)
postCond:
Example
<assertion1>
<condition2>
code<2,true>
<assertion2>
<assertion1>
<condition2>
code<2,false>
<assertion2>
88
Max( a,b,c )
preCond:
m=a
if( b>m )
m=b
endif
if( c>m )
m=c
endif
return(m)
postCond:
Example
<assertion2>
code
<postCond>
89
Loop Invariants
Any assumptions that
must be true about the
state of computation
every time it is at the
top of the loop.
Max( A )
preCond:
of n values.
i=1
m = A[1]
loop
exit when (i=n)
i=i+1
m = max(m,A[i])
endloop
return(m)
postCond:
m is max in A[1,n]
90
Loop Invariants
Any assumptions that
must be true about the
state of computation
every time it is at the
top of the loop.
Max( A )
preCond:
of n values.
i=1
m = A[1]
loop
loop-invariant:
m is max in A[1,i]
exit when (i=n)
i=i+1
m = max(m,A[i])
endloop
return(m)
postCond:
m is max in A[1,n] 91
Loop Invariants
Not just to please the prof.
Not just to prove correctness.
Use them to think about your
algorithm!
Before you start coding.
What do you want to be
true in the middle of your
computation?
You need to know.
Let your reader know.
Max( A )
preCond:
of n values.
i=1
m = A[1]
loop
loop-invariant:
m is max in A[1,i]
exit when (i=n)
i=i+1
m = max(m,A[i])
endloop
return(m)
postCond:
m is max in A[1,n] 92
Loop Invariants
<preCond>
codeA
loop
<loop-invariant>
exit when <exit Cond>
codeB
endloop
codeC
<postCond>
93
Loop Invariants
<preCond>
codeA
loop
<loop-invariant>
exit when <exit Cond> Definition of
codeB
<preCond>
endloop
<any conditions>
codeC
code
<postCond>
Correctness
<postCond>
How is this proved?

We dont know how many
times it will iterate.
94
Max( A )
preCond:
of n values.
i=1
m = A[1]
loop
loop-invariant:
m is max in A[1,i]
exit when (i=n)
i=i+1
m = max(m,A[i])
endloop
return(m)
postCond:
m is max in A[1,n]
Loop Invariants
Example
Step 0
<preCond>
codeA
<loop-inv>
95
Max( A )
preCond:
of n values.
i=1
m = A[1]
loop
loop-invariant:
m is max in A[1,i]
exit when (i=n)
i=i+1
m = max(m,A[i])
endloop
return(m)
postCond:
m is max in A[1,n]
Loop Invariants
Example
Step 1 Step 2 Step i

<loop-invariant>
<exit Cond>
codeB
<loop-inv>
96
Max( A )
preCond:
of n values.
i=1
m = A[1]
loop
loop-invariant:
m is max in A[1,i]
exit when (i=n)
i=i+1
m = max(m,A[i])
endloop
return(m)
postCond:
m is max in A[1,n]
Loop Invariants
Example
Last Step
<loop-invariant>
<postCond>
<exit Cond>
codeC
97
Partial Correctness

<preCond>
codeA
<loop-invariant>
Maintaining Loop Invariant

Exit
<loop-invariant>
<exit Cond>
codeB
<loop-invariant>
Clean up loose ends

<loop-invariant>
<postCond>
<exit Cond>
Exit
codeC
Proves that IF the program terminates then it works
<PreCond> & <code> <PostCond>
98
Algorithm Termination
You need to define some
Measure of progress
to prove that your algorithm
eventually terminates.
99
Simple Example
Insertion Sort Algorithm
100
Reconsidering Simple Examples

A good martial arts student will attentively repeat each
fundamental technique many times.
In contrast, many college students tune out when a concept
(e.g., depth first search) is taught more than once.
The better students listen carefully in order to refine and
develop their understanding.
Repetition Is An Opportunity
101
Code
class InsertionSortAlgorithm extends SortAlgorithm {
void sort(int a[]) throws Exception {
for (int i = 1; i < a.length; i++) {
int j = i;
int B = a[i];
while ((j > 0) && (a[j-1] > B)) {
a[j] = a[j-1];
j--; }
a[j] = B;
}}
102

5 km
9 km
103
Define Problem
Define Loop
Invariants
Define Measure of
Progress
79 km
to school
Define Step

Exit
Exit
Make Progress
Initial Conditions
Ending
Exit
79 km
75 km
km
0 km
Exit
Exit
104
Problem Specification
Precondition: The input is a list of n values
with the same value possibly repeated.
Post condition: The output is a list consisting
of the same n values in non-decreasing order.
88 52
14
31
25 98
30
23
62
79
14,23,25,30,31,52,62,79,88,98
105
State of computation determined
by values of all variables.
106
State of computation determined
by values of all variables.
88 52 14
31 25 62 30
23
98
79
14,23,25,30,31,52,62,79,88,98
88 14<52
31 25<62 30
23
79<98
107
Computation Stuck
Location too confusing for algorithm
Algorithm too lazy to define step for
every location.
88 52 14
31 25 62 30
23
98
79
14,23,25,30,31,52,62,79,88,98
88 14<52
31 25<62 30
23
79<98
108
Safe Locations
Algorithm specifies from which locations
it knows how to step.
88 52 14
31 25 62 30
23
98
79
14,23,25,30,31,52,62,79,88,98
109
Define Loop Invariant

Some subset of the elements are sorted
The remaining elements are off to the side.
88 52 14
31 25 62 30
23
98
79
14,23,25,30,31,52,62,79,88,98
23,31,52,88
30 14
25 62
98
79
110
Location Not Determined

Which subset of the elements are sorted, is
not specified.
14,52,62,79
30 23
25 88
98
31
23,31,52,88
30 14
25 62
98
79
111
Defining Measure of Progress

23,31,52,88
30 14
25 62
98
79
6 elements
to school
112
Define Step
23,31,52,88
30 14
25 62
98
79
113
Define Step
Select arbitrary element from side.
Insert it where it belongs.
23,31,52,88
30 14
25 62
98
79
23,31,52,62,88
30 14
25
98
79
114
Making progress while

Maintaining the loop invariant
23,31,52,88
Exit
79 km
75 km
30 14
25 62
98
79
6 elements
23,31,52,62,88
to school
Sorted sub-list
30 14
25
98
79
5 elements
to school
115
Beginning &
Ending
88 52 14
31 25 6 30
23
98
2
79
km
88 52 14
31 25 6 30
23
98
2
79
0 km
Exit
Exit
n elements
to school
0 elements
14,23,25,30,31,52,62,79,88,98
to school
14,23,25,30,31,52,62,79,88,9
116
Running Time
Inserting an element into a list of size i
takes (i) time.
Total = 1+2+3++n
=?
23,31,52,88
30 14
25 62
98
79
23,31,52,62,88
30 14
25
98
79
117
Adding Made Easy

1
...
n-1 +
= number of white dots
2........n
118
Adding Made Easy

1
n-1 +
...
n-1 +
n-2 +
...
= number of yellow dots
n ....... 2 1
2........n
119
Adding Made Easy

1
n-1 +
...
n-1 +
n-2 +
...
= number of yellow dots
n
n
2S dots
= n(n+1) dots in the
grid
n
n
n
S=
n (n+1)
2
n
n+1 n+1 n+1
n+1 n+1
120
Adding Made Easy
121
Adding Made Easy
122
Running Time
Inserting an element into a list of size i
takes (i) time.
Total = 1+2+3++n
= (n2)
23,31,52,88
30 14
25 62
98
79
23,31,52,62,88
30 14
25
98
79
123
Algorithm Definition Completed

Define Problem
Define Loop
Invariants
Define Measure of
Progress
79 km
to school
Define Step

Exit
Exit
Make Progress
Initial Conditions
Ending
Exit
79 km
75 km
km
0 km
Exit
Exit
124
Yet Another Abstract View

A Relay Race
T+
0
i-1
Your job is only to

Trust who passes you the baton
Go around once
i
Pass on the baton
125
Person 0: Carries baton region to safe region

<preCond>
codeA
<loop-invariant>
<preCond>
codeA
loop
<loop-invariant>
codeB
endloop
codeC
<postCond>
126
Person i: Running around the track

Exit
i-1
<loop-invariant>
<loop-invariant>
<exit Cond>
codeB
<preCond>
codeA
i
loop
<loop-invariant>
codeB
endloop
codeC
<postCond>
127
Last Person: Carries baton from safe region to finish.
Clean up loose ends

Exit
<loop-invariant>
<postCond>
<exit Cond>
codeC
<preCond>
1
T+
codeA
loop
<loop-invariant>
codeB
endloop
codeC
<postCond>
128
Insertion Sort
as a Relay Race
88 52 14
31 25 62 30
23
98
79
129
Establish Loop Invariant

from preconditions
88 52 14
31 25 62 30
23
98
79
130
88 52 14
31 25 62 30
23
98
79
131
i-1
31
88 52 14
25 62 30
23
98
79
132
31
88 52 14
25 62 30
23
98
79
133
i-1
31,88
52 14
25 62 30
23
98
79
134
31,88
52 14
25 62 30
23
98
79
135
i-1
25,31,88
52 14
62 30
23
98
79
136
25,31,88
52 14
62 30
23
98
79
137
i-1
25,31,52,88
14
62 30
23
98
79
138
14,23,25,30,31,52,62,79,88,98
139
Clean up loose ends
1
T+
14,23,25,30,31,52,62,79,88,98
140
Ok
I know you knew Insertion Sort
But hopefully you are beginning to appreciate
Loop Invariants
for describing algorithms
141
Explaining Insertion Sort

We maintain a subset of elements sorted
within a list. The remaining elements are
off to the side somewhere. Initially,
think of the first element in the array as
a sorted list of length one. One at a time,
we take one of the elements that is off to
the side and we insert it into the sorted
list where it belongs. This gives a sorted
list that is one element longer than it was
before. When the last element has been
inserted, the array is completely sorted.
142
Explaining Selection Sort

We maintain that the k smallest of the elements
are sorted in a list. The larger elements are in a set
on the side. Initially, with k=0, all the elements
are in this set. Progress is made by finding the
smallest element in the remaining set of large
elements and adding this selected element at the
end of the sorted list of elements. This increases k
by one. Stop with k=n. At this point, all the
elements have been selected and the list is sorted.
14,23,25,30,31
<
88 52
62
98
79
143
144
A Starry Night
How did Van Gogh come up with his famous
painting, A Starry Night?
There's no easy answer.
Designing algorithms is an art form.
145
Designing Loop Invariants

Coming up with the loop invariant is the
hardest part of designing an algorithm.
It requires practice, perseverance, and insight.
Yet from it
the rest of the algorithm
follows easily
146
Use This Process

Don't come up with the loop invariant after
the fact to make me happy.
Use it to design your algorithm.
147
Dont start coding

You must design a working algorithm first.
148
Exemplification:
Try solving the problem
on small input examples.
149
Start with Small Steps

What basic steps might you follow to make
some kind of progress towards the answer?
Describe or draw a picture of what the data
structure might look like after a number of
these steps.
150
Picture from the Middle

Leap into the middle of the algorithm.
What would you like your data structure to
look like when you are half done?
151
The Work Completed

The loop invariant should state what work
has been completed towards solving the
problem and what works still needs to be
done.
79 km
75 km
to school
to school
152
Flow Smoothly
The loop invariant should flow smoothly from

the beginning to the end of the algorithm.
At the beginning, it should follow easily from
the preconditions.
It should progress in small natural steps.
Once the exit condition has been met, the
postconditions should easily follow.
153
Ask for 100%

A good philosophy in life is to ask for 100%
of what you want,
but not to assume that you will get it.
What would you like to be true in the middle of
your computation?
154
Ask for 100%

Pretend that a genie has granted your wish.
You are now in the middle of your computation
and your dream loop invariant is true.
155
Ask for 100%

Maintain the Loop Invariant:
From here, are you able to take some
computational steps that will make progress
while maintaining the loop invariant?
156
Ask for 100%

If you can maintain the loop invariant, great.
If not,
Too Weak: If your loop invariant is too weak, then the
genie has not provided you with everything you need to
move on.
Too Strong: If your loop invariant is too strong, then
you will not be able to establish it initially or maintain
it.
157
No Assumptions:
Suppose a Martian jumped
into the top of the loop.
All you would know is that <loop-invariant> was true.
It alone must provide enough information
so that, after going around the loop, you can establish
that the loop invariant is again true.
158
Know What a LI Is Not

``the LI is ...''
code
A precondition
A postcondition
A statement that is always true
Eg: 1+1=2
The steps taken by the algorithm
159
Differentiating between Iterations

x=x+2
Meaningful as code
False as a mathematical statement
x = xi = value at the beginning of the iteration

x = xi+1 = new value after going around the
loop one more time.
x = x+2
Meaningful as a mathematical statement
160
Find Errors
Which of the steps

to develop an iterative
algorithm did the student
fail to do correctly?
161
Find Errors
If I = -5, s = 1-5 j = 0
Fine: Describes input

and output.
162
Find Errors
Variables need to
be documented.
Fine: Purpose outlined
in LI
163
Find Errors
Loop Invariants are pictures
of current state.
Not actions!
Not algorithms!
164
Find Errors
Fine: Shows a relationship

between the variables.
165
Find Errors
Let s and i be
values of s and i
when at the top of the loop.
Let s and i be
values after going
around again.
166
Find Errors
LI
s = j=1i j.
Code s = s+i
i = i+1.
Together:
s = (j=1i j) + i.
i is added in twice.
i should be added.
167
Find Errors
i = 1 j=1i j = 1 = s.
168
Find Errors
Exit
exit when i>I
Better to say:
loop
loop-invariant: _____
exit when i > I
169
Find Errors
Exit
exit when i>I
LI s = j=1i j.
Exit i > I
Together:
s = j=1I+1 j.
Post Cond.
170
Find Errors
exit when i>I
Test:
on typical value
i = 3 s=1+2+3.
on special values
i = 1 s=1.
i = 0 s=1.
171
Find Errors
PreCond: i=0
PostCond: i=131
Loop Invariant:
i is increasing
Step: i = i+2
Exit Cond: i=131
LI is an action not a picture.
172
Find Errors
Loop Invariant:
??
Step: i = i+2
Exit Cond: i=131
PreCond: i=0
PostCond: i=131
<preCond>
codeA
<loop-invariant>
Trivially true because

LI has no real content.
173
Find Errors
Loop Invariant:
??
Step: i = i+2
Exit Cond: i=131
PreCond: i=0
PostCond: i=131
Exit
<loop-invariant>
<exit Cond>
codeB
<loop-invariant>

LI has no real content.
174
Find Errors
PreCond: i=0
PostCond: i=131
Loop Invariant:
??
Step: i = i+2
Exit Cond: i=131
Clean up loose ends

Exit
<loop-invariant>
<exit Cond>
codeC
<postCond>

<exit Cond> = <postCond>
NEVER DO THIS!
175
Find Errors
Loop Invariant:
??
Step: i = i+2
Exit Cond: i=131
PreCond: i=0
PostCond: i=131
79 km
Exit
79 km
75 km
km
to school
Trivially true
Let measure be the value of i
(or 131-i)
176
Find Errors
PreCond: i=0
PostCond: i=131
0 km
Exit
Loop Invariant:
??
Step: i = i+2
Exit Cond: i=131
Sufficient progress is made

but alg fails to exit!
177
Find Errors
PreCond: i=0
PostCond: i=131
Loop Invariant:
??
Step: i = i+2
Exit Cond: i=131
0 km
Exit
This change ensures we exit.
178
Find Errors
PreCond: i=0
PostCond: i=131
Loop Invariant:
?? i is odd
Step: i = i+2
Exit Cond: i131
179
Find Errors
Loop Invariant:
i is odd
Step: i = i+2
Exit Cond: i131
PreCond: i=0
PostCond: i=131
<preCond>
codeA
i=0
?
i = i+1
i is odd
<loop-invariant>
180
Find Errors
PreCond: i=0
PostCond: i=131
Loop Invariant:
i is odd & i 132
Step: i = i+2
Exit Cond: i131
Clean up loose ends

Exit
<loop-invariant>
<exit Cond>
codeC
i is odd
i131
& i 132
i=131
<postCond>
181
Find Errors
Loop Invariant:
i is odd & i 132
Step: i = i+2
Exit Cond: i131
PreCond: i=0
PostCond: i=131
Exit
<loop-invariant>
<exit Cond>
codeB
i is even and 132

i < 131
i = i+2
i is even and 132

<loop-invariant> 182

Define Problem
Define Loop
Invariants
Define Measure of
Progress
79 km
to school
Define Step

Exit
Exit
Make Progress
Initial Conditions
Ending
Exit
79 km
75 km
km
0 km
Exit
Exit
183
Typical Types of Loop Invariants
More of the input

More of the output
Narrowed the search space
Case Analysis
Work Done
184
More of the Input

Loop Invariant
The input consists of an array of objects

Solution
Extra
We have read in the first i objects.

We will pretend that this prefix
is the entire input.
We have a solution for this prefix
plus some additional information
185
Loop Invariants and

Deterministic Finite Automaton
L = { {0,1}* | has length at most three and the

number of 1's is odd }.
186
More of the Input

Loop Invariant

Solution
Solution
Extra
Extra
We read in the i+1st object.

We will pretend that this larger prefix
We extend the solution we have
to one for this larger for prefix.
(plus some additional information)
187
More of the Input

Loop Invariant

Solution
Solution
Exit
79 km
75 km
Extra
Extra
i to i+1
188
More of the Input

Loop Invariant
Solution
Exit
End the end, we have read in the entire input.

The LI gives us that we have a solution
for this entire input.
189
Insertion Sort
The input consists of an array of integers
52,23,88,31,25,30,98,62,14,79
23,31,52,88
Solution

We will pretend that this prefix
We have a solution for this prefix
190
Insertion Sort
The input consists of an array of integers
52,23,88,31,25,30,98,62,14,79
23,31,52,88
23,25,31,52,88
We read in the i+1st object.

We will pretend that this larger prefix
We extend the solution we have
to one for this larger for prefix.
191
More of the Input

Loop Invariant
A student midterm answer:

Each iteration considers the next input item.
Loop Invariants are pictures
of current state.
Not actions!
Not algorithms!
192
More of the Input

Loop Invariant

We have considered the ith input element.
What about elements [1...i]?
193
More of the Input

Loop Invariant

We have considered input elements [1..i].
So?
194
More of the Input

Loop Invariant

Solution
We have a solution for each.
Each object does not need a separate solution
The input as a whole needs a solution.
195
More of the Input

Loop Invariant

Extra
Solution
We have a solution for the
prefix consisting of elements [1..i].
(plus some additional information)
196
Longest Block of Ones

00101111001100011111000011001
Alg reads the digits one at a time and remembers
enough about what read so that it does not need
to reread anything. (i.e. a DFA)
197

00101111001100011 111
When it has read this much,
what does it remember?
Largest block so far.
Read the next character &
re-determine the largest block so far.
198

00101111001100011 111
When it has read this much,
what does it remember?
Largest block so far.
Size of current block.
Read the next character &
re-determine the largest block so far
& current largest.
199
More of the input

More of the output
Case Analysis
Work Done
200
More of the Output

Loop Invariant
The output consists of an array of objects
I have produced the first i objects.
Produce the i+1st output object.
Exit
79 km
75 km
i to i+1
Exit
Done when
output n objects.
201
Input:
Selection Sort
52,23,88,31,25,30,98,62,14,79
The output consists of an array of integers

14,23,25,30,31,52,62,79,88,98
I have produced the first i objects.

Produce the i+1st output object.
Exit
Done when
output n objects.
202
More of the input

More of the output
Case Analysis
Work Done
Three
Binary Search like
examples
203
Define Problem: Binary Search

PreConditions
Key
25
Sorted List
3
13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
PostConditions
Find key in list (if there).
3
13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
204

Maintain a sub-list
Such that
key 25
3
13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
205

Maintain a sub-list.
If the key is contained in the original list,
then the key is contained in the sub-list.
key 25
3
13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
206
Define Step
Make Progress
key 25
3
13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
207
Define Step
Cut sub-list in half.
Determine which half key would be in.
Keep that half.
key 25
3
13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
208
Define Step
Determine which half the key would be in.
Keep that half.
key 25
3
13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
If key mid,
then key is in
left half.
If key > mid,

then key is in
right half.
209
Define Step
It is faster not to check if the middle
element is the key.
Simply continue.
key 43
3
13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
If key mid,
then key is in
left half.
If key > mid,

then key is in
right half.
210
Exit
Make Progress
79 km
75 km
The size of the list becomes smaller.

3
13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
79 km
13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
75 km
211
Initial Conditions
km
key 25
3
13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
n km
The sub-list is the

entire original list.
If the key is contained in

the original list,
then the key is contained
in the sub-list.
212
Ending Algorithm
Exit
key 25
3
13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
0 km
If the key is contained in the

original list,
then the key is contained in the
sub-list.
Sub-list contains one element.
Exit
If the key is
contained in the
original list,
then the key is at
this location.
213
If key not in original list

If the key is contained in
the original list,
then the key is contained
in the sub-list.
Loop invariant true,

even if the key is not
in the list.
key 24
3
13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
If the key is
contained in the
original list,
then the key is at
this location.
Conclusion still solves

the problem.
Simply check this one
location for the key.
214
Running Time
The sub-list is of size n, n/2, n/4, n/8,,1
Each step (1) time.
Total = (log n)
key 25
3
13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
If key mid,
then key is in
left half.
If key > mid,

then key is in
right half.
215
Code
216

Define Problem
Define Loop
Invariants
Define Measure of
Progress
79 km
to school
Define Step

Exit
Exit
Make Progress
Initial Conditions
Ending
Exit
79 km
75 km
km
0 km
Exit
Exit
217
Study:
Many experienced programmers were asked
to code up binary search.
80% got it wrong
Good thing is was not for a
nuclear power plant.
218
Make Precise Definitions

Maintain a sub-list with end points i & j.
key 25
3
13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
219

Maintain a sub-list with end points i & j.
key 25
3
13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
j
Does not matter which,
but you need to be consistent.
220

If the sub-list has even length,
which element is taken to be mid?
key 25
3
13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
i j
i j
mid
or mid
?
2
2
221

If the sub-list has even length,
which element is taken to be mid?
key 25
3
mid
13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
Should not matter

Choose right.
i j
2
mid
222
Common Bugs
Keep that half.
key 25
3
13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
If key mid,
then key is in
left half.
If key > mid,

then key is in
right half.
223
Common Bugs
If the middle element is the key,
it can be skipped over.
Exit
key 43
3
13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
If key mid,
then key is in
left half.
If key > mid,

then key is in
right half.
224
Common Bugs
Fix the bug.
key 43
3
13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
If key mid,
then key is in
left half.
If key > mid,

then key is in
right half.
225
Common Bugs
Second fix,
by making the left half slightly bigger.
New bug?
key 43
3
13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
If key mid,
then key is in
left half.
If key > mid,

then key is in
right half.
226
Exit
Common Bugs
79 km
75 km
Second fix,
by making the left half slightly bigger.
New bug?
key 43
3
13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
No progress is made.
Loop for ever!
227
Why is Binary Search so Easy to Get Wrong?
i j
1 mid
2
2 if key L(mid)
3
i = mid
else
j = mid -1
i j
OR
?
2
How many
possible
algorithms?
OR > ?
How many
correct
algorithms?
i = mid + 1
OR else
j = mid
Probability of
success by
guessing?
228
Moral
Use the loop invariant method to think
about algorithms.
Be careful with your definitions.
Be sure that the loop invariant is always
maintained.
Be sure progress is always made.
229
Loop Invariants
for
A second
Binary Search like
example
230
Define Problem: Binary Search Tree
<preCond>
Key
25
A binary search tree.
38
25
51
17
31
21
28
42
35
40
63
49
55
71
<postCond>
Find key in BST (if there).
231
Binary Search Tree

Its left children Any node Its right children
38
25
17
51
31
21
28
42
35
40
63
49
55
71
232

Maintain a sub-tree.
If the key is contained in the original tree,
then the key is contained in the sub-tree.
38
key 17
25
51
17
31
21
28
42
35
40
63
49
55
71
233
Define Step
Cut sub-tree in half.
Keep that half.
38
key 17
25
51
17
If key < root,

then key is
in left half.
31
21
28
If key = root,
then key is
found
42
35
40
63
49
55
71
If key > root,

then key is
in right half.
234

Define Problem
Define Loop
Invariants
Define Measure of
Progress
79 km
to school
Define Step

Exit
Exit
Make Progress
Initial Conditions
Ending
Exit
79 km
75 km
km
0 km
Exit
Exit
235
Loop Invariants
for
A third
Binary Search like
example
236
Card Trick
A volunteer, please.
237
Pick a Card
Done
238
Loop Invariant:
The selected card is one of
these.
239
Which column?
left
240
Loop Invariant:
these.
241
Selected column is placed

in the middle.
242
I will rearrange the cards.
243
Relax Loop Invariant:

I will remember the same
about each column.
244
Which column?
right
245
Loop Invariant:
these.
246

in the middle.
247
I will rearrange the cards.
248
Which column?
left
249
Loop Invariant:
these.
250

in the middle.
251
Here is your card.
Wow!
252
The Partitioning Problem

(used in quicksort)
Input:
Output:
p=52
88 52
14
31
25 98
30
23
62
79
14
31 30 23
25
88
52
62
98
79
253

p=52
31 23
25 98 30 14 62 79 88
or
p=52
31 23 71 25 98 30
p
62 79 88
p
254
Defining Measure of Progress

p=52
31 23
25 98 30 14 62 79 88
4 elements
not
looked at
255
Define Step
Make Progress
p=52
31 23
p
25 98 30 14 62 79 88
p
256
Define Step
Make Progress
p=52
31 23
25 98 30 14 62 79 88
p=52
31 23 14 25 98 30
p
62 79 88
p
257
Define Step
Four cases
p=52
31 23
p=52
25 98 30 14 62 79 88
31 23 14 25 98 30
62 79 88
p=52
31 23 14 25 98 30
62 79 88
31 23 14 25 98 30
62 79 88
p=52
31 23
25 98 30 71 62 79 88
31 23 71 25 98 30
31 23
25 98 30 71 62 79 88
31 23
62 79 88
25 98 30 71 62 79 88
258
Beginning &
Ending
p=52
88 52 14
31
25 62 30
23
98
79
km
0 km
Exit
Exit
p=52
88 25 31 98 62 14 30 79 23
n-1 elements
not
looked at
p=52
23 30 25 31 14
62 98 79 88
0 elements
not
looked at
25
23 31 14
30
62
52
98
79
88
259
Running Time
Each iteration takes (1) time.
Total = (n)
p=52
31 23
25 98 30 14 62 79 88
p=52
31 23 14 25 98 30
p
62 79 88
p
260

Define Problem
Define Loop
Invariants
Define Measure of
Progress
79 km
to school
Define Step

Exit
Exit
Make Progress
Initial Conditions
Ending
Exit
79 km
75 km
km
0 km
Exit
Exit
261
31 88 25 52 98 62 14 30 79 23
p=52
88 25 31 98 62 14 30 79 23
23 88 25 31 98 62 14 30 79
Tracing an
Example
23
25 31 98 62 14 30 79 88
23
25 31 98 62 14 30 79 88
23 30 25 31 98 62 14
79 88
23 30 25 31 98 62 14
79 88
23 30 25 31 98 62 14
79 88
23 30 25 31
62 14 98 79 88
23 30 25 31 14 62
23 30 25 31 14
98 79 88
62 98 79 88
262
Greatest Common Divisors

GCD
263
GCD(a,b)
= <64,44>
Input: <a,b>
Output: GCD(a,b) = 4
<a,b> <a-b,b>
GCD(a,b) = GCD(a-b,b)
GCD(64,44) = GCD(20,44) = 4
Maintain values <x,y> such that
GCD(a,b) = GCD(x,y)
<x,y> <x-y,y>
GCD(a,b) = GCD(x,y)
= GCD(x-y,y) = GCD(x,y)
Exit
79 km
75 km
smaller
264
GCD(a,b)
Input: <a,b> = <64,44>
<x,y> = <64,44>
= <20,44>
GCD(a,b) = GCD(x,y)
= <44,20>
= <24,20>
= < 4,20>
= <20, 4>
= <16, 4>
= <12, 4>
Running time?
= < 8, 4>
= < 4, 4>
= < 0, 4>
GCD(a,b) = GCD(x,y) = 4
265
GCD(a,b)
Input: <a,b> = <9999999999999,2>
<x,y> = <9999999999999,2>
= <9999999999997,2>
= <9999999999995,2>
= <9999999999993,2>
= <9999999999991,2>
Time = O(a)
Poly Time?
266
Time Complexity Is a Function

Specifies how the running time depends on
the size of the input.
A function mapping
size of input
time T(n) executed .
267
Definition of Time
# of seconds (machine dependent).
# lines of code executed.
# of times a specific operation is performed
(e.g., addition).
Which?
268
Definition of Time
# of seconds (machine dependent).
# lines of code executed.
# of times a specific operation is performed
(e.g., addition).
These are all reasonable definitions of time,
because they are within a constant factor of
each other.
269
Size of Input Instance?

83920
270
Size of Input Instance

5
83920
Size of paper
# of bits
# of digits
Value
- n = 2 in2
- n = 17 bits
- n = 5 digits
- n = 83920
Which are reasonable?
271

5
83920
Size of paper
# of bits
# of digits
Value
- n = 2 in2
- n = 17 bits
- n = 5 digits
- n = 83920
Intuitive
272

5
83920
Size of paper
# of bits
# of digits
Value
- n = 2 in2
- n = 17 bits
- n = 5 digits
- n = 83920
Intuitive
Formal
273

5
83920
Size of paper
# of bits
# of digits
Value
- n = 2 in2
- n = 17 bits
- n = 5 digits
- n = 83920
Intuitive
Formal
Reasonable
# of bits =
3.32 * # of digits
274

5
83920
Size of paper
# of bits
# of digits
Value
- n = 2 in2
- n = 17 bits
- n = 5 digits
- n = 83920
Intuitive
Formal
Reasonable
Unreasonable
# of bits = log2(Value)
Value = 2# of bits
275
GCD(a,b)
Input: <a,b> = <9999999999999,2>
<x,y> = <9999999999999,2>
= <9999999999997,2>
= <9999999999995,2>
= <9999999999993,2>
= <9999999999991,2>
Time = O(a) = 2O(n)
Size = number of digits n = O(log(a))
276
GCD(a,b)
Speedup
<x,y> <x-y,y>
<x-2y,y>
<x-3y,y>
<x-4y,y>
<x-iy,y>
<remainder with x/y,y>
= <x mod y,y>
<y,x mod y> smaller than y
277
GCD(a,b)
Input: <a,b> = <44,64>
<x,y> = <44,64>
GCD(a,b) = GCD(x,y)
= <64,44>
<x,y> <y,x mod y>
= <44,20>
= <20, 4>
= < 4, 0>
278
GCD(a,b)
Input: <a,b> = <10000000000001,9999999999999>

<x,y> = <10000000000001,9999999999999>
= <9999999999999,2>
Little progress
= <2,1>
Lots of progress
= <1,0>
Every two iterations:
the value x decreases by at least a factor of 2.
the size of x decreases by at least one bit
Time = O(log(a)+log(b)) = O(n)
Size = n = O(log(a)+log(b))
279
GCD(a,b)
280

Input: A pile of things to sort.
(Ex: Exams)
Output: Sort them
Requirements:
Easy to execute by humans
Only can have 7 piles on desk
Can move one exam (or pile)
at a time.
Fast O(nlog(n)) time.
Likely the only algorithm
in this course which you will
execute by hand yourself.
281

Input:
[A-Z]
Denotes an unsorted pile of exams

with last names starting in the range [A-Z]
282

[A-Z]
Psychology study:
Humans can only think about
5 things at once.
Is the first letter
of the name
on the first exam
in
[A-E]
[F-K]
[L-O]
[P-T]
[U-Z]
283

[A-Z]
[A-E] [F-K] [L-O] [P-T] [U-Z]
Bucket Sort
Put exams one at a time
in the correct bucket.
284

[]
sorted
[A-E]
[F-K]
[L-O]
[P-T]
[U-Z]
A sorted pile (upside down)

before all the rest
A stack of piles
each pile unsorted
all in one pile before all in next
285

[A-E]
[F-K]
[L-O]
[P-T]
[U-Z]
[]
sorted
[A]
[B]
[C]
[D]
Bucket Sort
Put exams one at a time
in the correct bucket.
[E]
286

[A]
[]
[B]
[C]
[D]
sorted
[E]
[F-K]
[L-O]
[P-T]
[U-Z]
before all the rest
A stack of piles
each pile unsorted
287

[]
sorted
[A]
[B]
[C]
[D]
[E]
[F-K]
[L-O]
[P-T]
[U-Z]
[AA-AE][AF-AK][AL-AO] [AP-AT] [AU-AZ]
288

[]
[AU-AZ]
[B]
sorted
[AA-AE]
[E]
[F-K]
[U-Z]

before all the rest
A stack of piles
each pile unsorted
289
[AA-AE]
[]
[AU-AZ]
[B]
sorted
[E]
[F-K]
[AA-AE]
[U-Z]
290
[AF-AK]
[]
[AU-AZ]
[B]
sorted
[E]
[F-K]
[AA-AE]
When sufficiently small
sort by hand
[U-Z]
sorted
291

[AA-AE]
[AU-AZ]
[B]
sorted
[AF-AK]
[E]
[F-K]
[U-Z]

before all the rest
A stack of piles
each pile unsorted
292
[AF-AK]
[AA-AE]
[AU-AZ]
[B]
sorted
[E]
[F-K]
[U-Z]
[AF-AK]
sorted
293

[AA-AK]
[AU-AZ]
[B]
sorted
[AL-AO]
[E]
[F-K]
[U-Z]

before all the rest
A stack of piles
each pile unsorted
294

[AA-AK]
[AU-AZ]
[B]
sorted
[AL-AO]
[E]
[F-K]
[U-Z]
[AL-AO]
[AP-AT]
[AU-AZ]
sorted
sorted
sorted
295

[AA-AZ]
[E]
[F-K]
sorted
[B]
[U-Z]
296

[A]
[E]
[F-K]
sorted
[B]
[U-Z]

before all the rest
A stack of piles
each pile unsorted
297

[B]
[A]
sorted
[E]
[F-K]
[U-Z]
[BA-BE][BF-BK][BL-BO] [BP-BT] [BU-BZ]
298

[A]
[BU-BZ]
[C]
sorted
[BA-BE]
[E]
[F-K]
[U-Z]

before all the rest
A stack of piles
each pile unsorted
299

[A-ZT]
[ZU-ZZ]
sorted
[ZU-ZZ]
sorted
300

[A-Z]
sorted
Exit
301
RadixSort
Input: A of stack of N punch cards.
Each card contains d digits.
Each digit between 0&(k-1)
Output: Sort the cards.
Bucket Sort Machine:
Select one digit
Separates cards into k piles.
wrt selected digit.
344
125
333
134
224
334
143
225
325
243
125
224
225
325
333
134
334
344
143
243
Stable sort: If two cards are the same for that digit,
their order does not change.
302
RadixSort
344
125
333
134
224
334
143
225
325
243
Sort wrt which

digit first?
The most
significant.
125
134
143
224
225
243
344
333
334
325
125
224
Sort wrt which 225
digit Second? 325
134
The next most 333
334
significant.
143
243
344
All meaning in first sort lost.
303
RadixSort
344
125
333
134
224
334
143
225
325
243
Sort wrt which

digit first?
The least
significant.
333
143
243
344
134
224
334
125
225
325
224
125
Sort wrt which 225
digit Second? 325
333
The next least 134
334
significant.
143
243
344
304
RadixSort
344
125
333
134
224
334
143
225
325
243
Sort wrt which

digit first?
The least
significant.
333
143
243
344
134
224
334
125
225
325
2 24
1 25
Sort wrt which 2 25
digit Second? 3 25
3 33
The next least 1 34
3 34
significant.
1 43
2 43
3 44
Is sorted wrt first i digits.
305
RadixSort
2 24
1 25
2 25
3 25
3 33
1 34
3 34
1 43
2 43
3 44
i+1
Is sorted wrt
first i digits.
Sort wrt i+1st

digit.
1 25
1 34
1 43
2 24
2 25
2 43
3 25
3 33
3 34
3 44
Is sorted wrt
first i+1 digits.
These are in the
correct order
because sorted
wrt high order digit.
306
RadixSort
2 24
1 25
2 25
3 25
3 33
1 34
3 34
1 43
2 43
3 44
i+1
Is sorted wrt
first i digits.
Sort wrt i+1st

digit.
1 25
1 34
1 43
2 24
2 25
2 43
3 25
3 33
3 34
3 44
Is sorted wrt
first i+1 digits.
These are in the
correct order
because was sorted &
stable sort left sorted.
307
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output: 0 0 0 0 0 1 1 1 1 1 1 1 1 1 2 2 3 3 3
Input: N records each labeled with a digit
between 0&(k-1).
Output: Stable sort the numbers.
Algorithm: Count to determine where records go.
Go through the records in order
putting them where they go.
308
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output: 0 0 0 0 0 1 1 1 1 1 1 1 1 1 2 2 2 3 3
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Stable sort: If two records are the same for that digit,
their order does not change.
Therefore, the 4th record in input with digit 1 must be
the 4th record in output with digit 1.
It belongs in output index 8, because
8 records go before it
ie 5 records with a smaller digit &
3 records with the same digit
Count
These309
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output:
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v:
# of records with digit v:
0
5
1
9
2
3
3
2
N records, k different values. Time to count? (N k)

We have counted # of each value (N)
in the first i values.
310
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output:
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v:
# of records with digit < v:
0
5
0
1
9
5
2 3
3 2
14 17
N records, k different values. Time to count? (k2)

Too much
311
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output:
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v:
0
5
0
1
9
5
2 3
3 2
14 17
Computed
N records, k different values. Time to count?
(k)
312
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output: 0 0 0 0 0 1 1 1 1 1 1 1 1 1 2 2 2 3 3
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v:
Location of first record
with digit v.
0
0
1
5
2 3
14 17
313
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
1
Output: 0 ?
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v:
Location of first record
with digit v.
0
0
1
5
2 3
14 17
Algorithm: Go through the records in order

314
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
1
Output:
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v:
Location of next record
with digit v.
0
0
1
5
2 3
14 17

315
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
1
Output: 0
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v:
with digit v.
0
0
1
6
2 3
14 17

316
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
1
Output: 0 0
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v:
with digit v.
0
1
1
6
2 3
14 17

317
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
1 1
Output: 0 0
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v:
with digit v.
0
2
1
6
2 3
14 17

318
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
1 1
Output: 0 0
3
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v:
with digit v.
0
2
1
7
2 3
14 17

319
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
1 1 1
3
Output: 0 0
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v:
with digit v.
0
2
1
7
2 3
14 18

320
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
1 1 1 1
3
Output: 0 0
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v:
with digit v.
0
2
1
8
2 3
14 18

321
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
1 1 1 1
3 3
Output: 0 0
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v:
with digit v.
0
2
1
9
2 3
14 18

322
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
1 1 1 1 1
3 3
Output: 0 0
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v:
with digit v.
0
2
1
9
2 3
14 19

323
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
1 1 1 1 1
3 3
Output: 0 0 0
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v:
with digit v.
0
2
1 2 3
10 14 19

324
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
1 1 1 1 1
3 3
Output: 0 0 0
2
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v:
with digit v.
0
3
1 2 3
10 14 19

325
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
1 1 1 1 1 1
2
3 3
Output: 0 0 0
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v:
with digit v.
0
3
1 2 3
10 15 19

326
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
1 1 1 1 1 1
2
3 3
Output: 0 0 0
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v:
with digit v.
0
3
1 2 3
10 15 19

327
CountingSort
Input: 1 0 0 1 3 1 1 3 1 0 2 1 0 1 1 2 2 1 0
Output: 0 0 0 0 0 1 1 1 1 1 1 1 1 1 2 2 2 3 3
Index: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Value v:
with digit v.
0
5
1 2 3
14 17 19

Time = (N)
(Counting Ops)
Total = (N+k) (log N + log k) (bit ops)
328
Radix/Counting Sort
Input: N numbers, each L bits long.
Output: Sort the numbers.
N
111101110100101000101
101001010100010110111
110100011010011110101
L
111 101 110 100 101 000 101

N 101 001 010 100 010 110 111
110 100 011 010 011 110 101
= log k (k will be set to N)
d = L / log k
329
Radix/Counting Sort
Input: N numbers, each L bits long.
Each card contains d digits.
Each digit between 0&(k-1)
Output: Sort the numbers.
Use Radix Sort: Sort wrt each digit using Counting Sort.
111 101 110 100 101 000 101

N 101 001 010 100 010 110 111 digit between 0&(k-1).
110 100 011 010 011 110 101
= log k
d = L / log k
330
Radix/Counting Sort
331
Radix/Counting Sort
Time = (Time of Radix Sort)
= (# of digits) (Time of Counting Sort)
= L/log k
(N+k) (log N + log k) bit ops
Set k to minimize time.
Wants k big.
Really wants k small, but does not

care as long as k N.
Set k=N
332
Radix/Counting Sort
Time = (Time of Radix Sort)
= (# of digits) (Time of Counting Sort)
= L/log k
(N+k) (log N + log k) bit ops
= L/log N N
log N bit ops
= (L N) bit ops = (n) bit ops
Size = Size of N numbers, each L bits long.
= (L N) = n
Linear Time
But sorting should take (N log N) time!!!
333
Radix/Counting Sort
Time = (L N) bit ops (N log N) bit ops
Size = Size of N numbers, each L bits long.
= (L N)
L => log N if you want N distinct numbers.
Merge or Quick Sort

Time = (N log N) comparisons = (n)
Size = Size of N numbers, each arbitrarily long.
= (N log N) bits.
334
A Lower Bound on Sorting

Merge, Quick, and Heap Sort can sort N numbers
using O(N log N) comparisons between the values.
Theorem: No algorithm can sort faster.
335

A, I, [ A(I) P(I) or Time(A,I) Tlower(|I|)]
I have an algorithm A that I
claim works and is fast.
Oh yeah, I have an input I
for which it does not .
I win if A on input I gives
the wrong output or
runs slow.
336

Sorting
Input:
Values:
Indexes:
23
67
34
21
87
13
11
Output:
Values:
Indexes:
11
13
21
23
34
64
87
Lower Bound:
Input: An Algorithm for Sorting A
Output: An instance I
on which alg A either
Values:
Indexes:
23
67
34
21
87
13
11
Takes too much time

Or give the wrong answer
Values:
Indexes:
11
13
21
23
34
64
87
337

Define Problem
Define Loop
Invariants
Define Measure of
Progress
79 km
to school
Define Step

Exit
Exit
Make Progress
Initial Conditions
Ending
Exit
79 km
75 km
km
0 km
Exit
Exit
338
I give you algorithm A

I claim it sorts.
I must output an instance

Values:
Indexes:
23
67
34
21
87
13
11

Takes too much time
It might as well be a
permutation of 1..N
(indexes not shown)
4
339

Need to know
what the algorithm does
before we can know
what input to give it.

I claim it sorts.
Break this cycle,

one iteration at a time.
Need to give the
algorithm an input
before we can know
what the algorithm does input.
340

Restrict the search space
341

I maintain a set of instances
(permutations of 1..n)
1
3
4
2
5
8
4
2
1
6
5
3
3
6
6
Oh dear!
The first t time steps of my
algorithm A are the same given
any of these instances!!!
342

Initially, I consider all N!
permutations of 1..N
1
3
4
2
5
8
4
2
1
6
5
3
3
6
6
Oh dear!
The first 0 time steps of my
343

Initially, I consider all N!
permutations of 1..N
1
3
4
2
5
79 km
km
8
4
2
1
6
5
3
3
6
6
The measure of progress

is the number of instances.
Initially, there are N! of them.
344
3
4
2
5
8
4
2
1
6
5
3
3
6
6
My t+1st time step

a7 < a2
5th bit of a7 + a2
Any yes/no
question
Because the first t time steps

are the same, what the alg
knows is the same.
Hence, its t+1st step is the same.
345

I partition my instances based
on what they do on this t+1st
step.
1
3
4
2
5
8
4
2
1
6
5
3
3
6
6
My t+1st time step

a7 < a2
5th bit of a7 + a2
Any yes/no
question
Because the first t time steps

are the same, what the alg
knows is the same.
Hence, its t+1st step is the same.
346

I keep the larger half.
1
3
4
2
5
8
4
2
1
6
5
3
3
6
6
347

1
4
3
5
2
6
5
7
8
8
6
Oh dear!
The first t+1 time steps of my
348

1
4
3
79 km
5
2
6
5
7
8
8
6
Initially, I have n! Permutations.

After t time steps I have N!/2t
349

1
4
3
Exit
5
2
6
5
7
8
8
6
Initially, I have n! Permutations.

After t time steps I have N!/2t
We exit when there are two
instances.
T = log(N!)
350
/2 factors each at least N/2.

N! = 1 2 3 N/2 N
N
N factors each at most N.

NN
Exit

instances.
T = log(N!) = (N log(N)).
351
Exit

I claim it sorts.
Values:
Indexes:
23
67
34
21
87
13
11

Takes too much time
352
Exit

Values:
Indexes:
23
67
34
21
87
13
11

Takes too much time
Case 1: The algorithm does not
stop at time T on these two
instances.
Exit
Oops!
T = log(N!) = (N log(N)).
353
Exit

Values:
Indexes:
23
67
34
21
87
13
11

Takes too much time
Exit
Oops!
I must give
the wrong
answer on one
of these!
Case 2: The algorithm stops at time

T and gives an answer.
instances
and these need different answers.

The first T time steps of alg A are
the same on these two instances
and hence the same answer is
354

Theorem: For every sorting algorithm A,
on the worst case input instance I,
(N log2 N) comparisons (or other bit operations)
need to be executed to sort N objects.
A, I, A(I) P(I) or Time(A,I) N log2 N

Proof: Prover/Adversary Game
355
End Iterative Algorithms

Math Review
356

01 Iterative Methods

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

01 Iterative Methods

Uploaded by

Copyright:

Available Formats

Iterative Algorithms

& Loop Invariants

Understanding Algorithms Abstractly

Pros and Cons?

Pros and Cons?

DFA and Turing Machines

Pros and Cons?

DFA and Turing Machines

Higher Level Abstract View

Software developers view subsystems as entities with

Higher Level Abstract View

View from which

Two Key Types of Algorithms

A good way to structure

I like understanding things

The Getting to School Problem

Maintain Loop Invariant

Maintain Loop Invariant

Maintain Loop Invariant

No. What if it is not initially true?

Establishing Loop Invariant

Maintain Loop Invariant

Can we be assured that the

Maintain Loop Invariant

Ending The Algorithm

Designing an Algorithm Is this sufficient?

Define Exit Condition Maintain Loop Inv

Loop Invariant Maintained

Computation can always proceed

Computation always makes progress

We will now redo

Is the black the form and the green the background?

If you are describing an alg to a

At least tell me what the algorithm

How can you possibly understand

How can you possibly understand

How is this proved?

Must check 2r different

How is this proved?

Step 1 Step 2 Step i

Establishing Loop Invariant

Maintaining Loop Invariant

Clean up loose ends

Reconsidering Simple Examples

Higher Level Abstract View

Define Exit Condition Maintain Loop Inv

Define Loop Invariant

Location Not Determined

Defining Measure of Progress

Making progress while

Adding Made Easy

= number of white dots

Adding Made Easy

= number of white dots

= number of yellow dots

Adding Made Easy

= number of white dots

= number of yellow dots

Adding Made Easy

Adding Made Easy

Algorithm Definition Completed

Define Exit Condition Maintain Loop Inv