asymptotic notation

Because it is so easy to cheat with the best case running time, we usually don't rely too much
about it.
Because it is usually very hard to compute the average running time, since we must somehow
average over all the instances, we usually strive to analyze the worst case running time.
The worst case is usually fairly easy to analyze and often close to the average or real running
time.
We have agreed that the best, worst, and average case complexity of an algorithm is a numerical
function of the size of the instances.

However, it is difficult to work with exactly because it is typically very complicated!
Thus it is usually cleaner and easier to talk about upper and lower bounds of the function.
This is where the dreaded big O notation comes in!
Since running our algorithm on a machine which is twice as fast will effect the running times by
a multiplicative constant of 2 - we are going to have to ignore constant factors anyway.
Introduction
A problem may have numerous algorithmic solutions. In order to choose the best algorithm for a
particular task, you need to be able to judge how long a particular solution will take to run. Or,
more accurately, you need to be able to judge how long two solutions will take to run, and
choose the better of the two. You don't need to know how many minutes and seconds they will
take, but you do need some way to compare algorithms against one another.
Asymptotic complexity is a way of expressing the main component of the cost of an algorithm,
using idealized units of computational work. Consider, for example, the algorithm for sorting a
deck of cards, which proceeds by repeatedly searching through the deck for the lowest card. The
asymptotic complexity of this algorithm is the square of the number of cards in the deck. This
quadratic behavior is the main term in the complexity formula, it says, e.g., if you double the size
of the deck, then the work is roughly quadrupled.
The exact formula for the cost is more complex, and contains more details than are needed to
understand the essential complexity of the algorithm. With our deck of cards, in the worst case, the
deck would start out reverse-sorted, so our scans would have to go all the way to the end. The first scan
would involve scanning 52 cards, the next would take 51, etc. So the cost formula is 52 + 51 + ... + 1.
generally, letting N be the number of cards, the formula is 1 + 2 + ... + N, which equals (N + 1) * (N / 2) =
(N
2
+ N) / 2 = (1 / 2)N
2
+ N / 2. But the N^2 term dominates the expression, and this is what is key for
comparing algorithm costs
Asymptotically speaking, in the limit as N tends towards infinity, 1 + 2 + ... + N gets closer and closer to
the pure quadratic function (1/2) N^2. And what difference does the constant factor of 1/2 make, at this
level of abstraction. So the behavior is said to be O(n
2
).
Now let us consider how we would go about comparing the complexity of two algorithms. Let
f(n) be the cost, in the worst case, of one algorithm, expressed as a function of the input size n,
and g(n) be the cost function for the other algorithm. E.g., for sorting algorithms, f(10) and g(10)
would be the maximum number of steps that the algorithms would take on a list of 10 items. If,
for all values of n >= 0, f(n) is less than or equal to g(n), then the algorithm with complexity
function f is strictly faster. But, generally speaking, our concern for computational cost is for the
cases with large inputs; so the comparison of f(n) and g(n) for small values of n is less significant
than the "long term" comparison of f(n) and g(n), for n larger than some threshold.
Big-O Notation
Definition
Big-O is the formal method of expressing the upper bound of an algorithm's running time. It's a
measure of the longest amount of time it could possibly take for the algorithm to complete.
More formally, for non-negative functions, f(n) and g(n), if there exists an integer n
0
and a
constant c > 0 such that for all integers n > n
0
, f(n) cg(n), then f(n) is Big O of g(n). This is
denoted as "f(n) = O(g(n))". If graphed, g(n) serves as an upper bound to the curve you are
analyzing, f(n).

We say
f(n) is order g(n)
f(n) is big-Oh of g(n)
Visually, this says that the f(n) curve must eventually fit under the cg(n) curve
Theory Examples
So, let's take an example of Big-O. Say that f(n) = 2n + 8, and g(n) = n
2
. Can we find a constant
c, so that 2n + 8 <= n
2
? The number 4 works here, giving us 16 <= 16. For any number c greater
than 4, this will still work. Since we're trying to generalize this for large values of n, and small
values (1, 2, 3) aren't that important, we can say that f(n) is generally faster than g(n); that is, f(n)
is bound by g(n), and will always be less than it.
It could then be said that f(n) runs in O(n
2
) time: "f-of-n runs in Big-O of n-squared time".
To find the upper bound - the Big-O time - assuming we know that f(n) is equal to (exactly) 2n +
8, we can take a few shortcuts. For example, we can remove all constants from the runtime;
eventually, at some value of c, they become irrelevant. This makes f(n) = 2n. Also, for
convenience of comparison, we remove constant multipliers; in this case, the 2. This makes f(n)
= n. It could also be said that f(n) runs in O(n) time; that lets us put a tighter (closer) upper bound
onto the estimate.
Examples:
Consider the function f(n)=8n+128. Clearly, f(n) is non-negative for all integers . We wish
to show that
According to Definition , in order to show this we need to find an integer and a constant
c>0 such that for all integers ,
It does not matter what the particular constants are--as long as they exist! For example, suppose
we choose c=1. Then

Since (n+8)>0 for all values of , we conclude that . That is, .
So, we have that for c=1 and , for all integers . Hence,
. Figure clearly shows that the function is greater than the function
f(n)=8n+128 to the right of n=16.



Advantage of O notation
1. It is possible to compare of two algorithms with running times
2. Constants can be ignored Units are not important
O(7n2) = O(n2)
3. Lower order terms are ignored Compare relative growth only
O(n3+7n2+3) = O(n3)
Running Times of Algorithm A and B
TA(N)= 1000 N = O(N)
TB(N)= N2 = O(N2) A is asymptotically faster than B !


Big-Oh Rules
1. If is f(n) a polynomial of degree d, then f(n) is O(n), i.e.,
a Drop lower-order terms
b Drop constant factors
2. Use the smallest possible class of functions
Say 2n is O(n) instead of 2n is O(n2)
3. Use the simplest expression of the class
Say 3n + 5 is O(n) instead of 3n + 5 is O(3n)


Review: The Difference Between T(N) and O(N)
T(n) : is total run time.
O(n) : is the approximation to the run time where we ignore constants and lower order terms. We call it
the growth rate. Also called asymptotic approximation.

Example:
if T(n) = 3 n2 + n + 1
then T(n) = O(n2)

if T(n) = 3 n log(n) + 2
then T(n) = O(n log(n))
Big-O Is Worst Case
Remember, Big-O says nothing about specific inputs, or specific input sizes.
Suppose I give Bubble Sort the list 1, 2, 3, 4, 5
Stops right away. Fast!
Suppose I give Bubble Sort the list 5, 4, 3, 2, 1
Worst case scenario! Slow.

So need to calculate the max number of times code goes through a loop.
e.g., if use a while loop, then the number of times code iterates should be calculated for the worst
case.

By the way, a while loop is just like a for loop when calculating run time and growth rates.
Predicting How Long To Run
Can predict how long it will take to run a program with a very large data set.
Do a test with a small practice data set.
Then use big-O to predict how long it will take with a real (large) data set.
Cool!
Suppose we are using a program that is O(N2).
Our test shows that it takes 1 minute to run 100 inputs. How long will it take to run 1000 inputs?
Set it up this way:

General O Rules: Rule 1
Rule 1
if T1(N) = O(f(N)) and T2(N) = O(g(N)) then
(a) T1(N) + T2(N) = max( O(f(N)), O(g(N)) ).
(b) T1(N) * T2(N) = O( f(N)*g(N) ).
Example
for(int i=1; i<= nNum; i++)
{
nPartialSum += i * i * i;
}
for(int i=1; i<= nNum; i++)
{
for(int j=1; j<= nNum; j++)
{
nPartialSum += i * j;
}
}
So the total run time is
T1(N) + T2(N) = max(O(N), O(N2)) = O(N2)
General O Rules: Rule 2
Rule 2
(logN)k = O(N) for any constant k.


( ) ( )
min
1000
min 1
100
2 2
x takes
inputs
takes
inputs
=
What?

Remember: T(N) = O(f(N)) when T(N) s cf(N).
So, were just saying that (logN)k grows more slowly than N.

In other words: logarithms grow VERY slowly.

11
Advantages of O Notation
- It is possible to compare of two algorithms with running times
- Constants can be ignored. Units are not important. O(7n2) = O(n2)
- Lower order terms are ignored Compare relative growth only
O(n3+7n2+3) = O(n3)

Running Times of Algorithm A and B
TA(N)= 1000 N = O(N)
TB(N)= N2 = O(N2)
A is asymptotically faster than B !





Omega notations
Just as O notation provides an asymptotic upper bound on a function, Omega notation provides
an asymptotic lower bound. Since Omega notation describes a lower bound, when we use it to
bind the best-case running time of an algorithm, by implication we also bound the running time of
the algorithm on arbitrary inputs as well.
Definition :
T(N)=_(f(N))
if there are positive constants c and n0 such that
T(N)cf(N) when N n0
T(N) grows no slower than f(N)
growth rate of T(N) is greater than or equal to
growth rate of f(N) for large N
f(N) is a lower bound on T(N)








Theorem:
f(N)=O(g(n)) . g(n)= (f(N))
Proof:
f(N).c1g(n) . g(n).c2f(N)
divide the left side with c1
1/c1 f(N).g(n) . g(n).c2f(N)

Theta Notations()
Definition :
T(N)=(f(N)) if and only if
T(N)=O(f(N)) and T(N)=_(f(N))
T(N) grows as fast as f(N)
growth rate of T(N)and f(N)are equal for large N
f(N)is a tight bound on T(N)








Example :
T(N) = 3N2
T(N) = O(N4)
T(N) = O(N3)
T(N) = (N2)-best

RULE 1:
if
T1(N)=O(f(N)) and T2(N)=O(g(N)) then
a) T1(N)+T2(N)= max(O(f(N)),O(g(N)))
b) T1(N)*T2(N)= O(f(N)*g(N))
RULE 2:
if T(N) is a polynomial of degree k
T(N)=akNk + ak-1Nk-1 + + a1N + a0
then
T(N)=O(Nk)
RULE 3:
logkN=o(N) for any constant k
logarithm grows very slowly !




problems
Give asymptotic upper bound for T(n) in each of the following recurrences:
a) T(N) = 3T(N / 3) +1 T (1) = 1
b) T(N) = 2T(N 1) + 2N T(0) = 1
c) T(N) = T(N 1) + N2 T(1) = 1
Prove or disprove the following
a) If T1(N) = (f(N)) and T2(N) = (g(N)) then
i. T1(N) + T2(N) = max ( (f(N)), (g(N)))
ii. T1(N) * T2(N) = (f(N) * g(N))
b) If T1(N) = _ (f(N)) and T2(N) = _ (g(N)) then
i. T1(N) + T2(N) = max (_ (f(N)), _ (g(N)))
ii. T1(N) * T2(N) = _ (f(N) * g(N))
4-

The maximum rule
A useful tool for proving that one function is in the order of another is the
maximum rule.
Let
f ,g :N Rc 0 be two arbitrary functions from the natural numbers to
the nonnegative reals. The maximum rule says that
SOFTENG 211 Asymptotic notation (2004)
- 5 -
O( f (n) + g(n)) = O(max( f (n),g(n))). More specifically, let

p,q : N Rc0 be
defined for each natural number n by p(n) = f(n)+g(n) and q(n) =
max(f(n),g(n)), and consider an arbitrary function

t : N Rc0 . The
maximum rule says that
t(n) O( p(n)) if and only if

t(n) O(q(n)).
The maximum rule also applies with
notation