Chapter 3  Growth of Functions           3.1.1


"Asymptotic" notation is used to answer the
question: 

   What is the LIMIT of an algorithm's
   running time when the size of its input
   increases WITHOUT BOUND?

This is the algorithm's asymptotic efficiency

3.1 Asymptotic notation

The functions we will use to measure
asymptotic efficiency are defined on the set
of natural numbers N = {0,1,2,...} -- for
example the worst-case running-time function
T(n).  Occasionally it will be convenient to
abuse the notation by extending it to real
numbers or restricting it to a subset of N
(the even numbers, for example).  But we must
be careful not to mis-use the notation.

Theta-notation                           3.1.2

We found that the worst-case running time of
insertion sort is T(n) = Theta(n^2).  Here is
the precise definition of that notation.

Definition: For a given function g(n), we
denote by Theta(g(n)) the set of functions:

Theta(g(n)) = { f : there exist positive
                    constants c1, c2, and n0
                    such that
                0 <= c1g(n) <= f(n) <= c2g(n)
                    for all n >= n0 }

So a function f belongs to Theta(g(n)) if
there exist positive constants c1 and c2 such
that f(n) can be "sandwiched" between c1g(n)
and c2g(n) for sufficiently large n.

Note that we should say that "f is a member of
Theta(g(n))", but it is almost always
expressed as "f(n) = Theta(g(n))", which may
be a bit confusing, but has advantages.

Figure 3.1(a) shows this situation: for all
n to the right of n0, the value of f(n) lies
at or above c1g(n) and at or below c2g(n).

In this case, we say g(n) is an 
"asymptotically tight" bound for f(n).

                                         3.1.3
 The definition of Theta(g(n)) requires that
each member function f of it be asymptotically
nonnegative, i.e. f(n) >= 0 whenever n is
sufficiently large.  (Similarly we say a
function f is asymptotically positive if
f(n) > 0 whenever n is sufficiently large.)

 Consequently g(n) must be asymptotically
nonnegative, or else Theta(g(n)) is empty.
We therefore assume that all functions used
in the Theta notation and other asymptotic
notations are asymptotically nonnegative.

 Previously we used an informal notation
for Theta that involved ignoring lower-order
terms and the coefficient of the highest-order
term.  We justify this technique for a
specific case -- we show that:

   1/2*n^2 - 3n = Theta(n^2)

We must find positive constants c1, c2, and n0
such that:

   c1*n^2 <= 1/2*n^2 - 3n <= c2*n^2

for all n >= n0.  Dividing by n^2 gives:

   c1 <= 1/2 - 3/n <= c2

The right inequality holds if c2 = 1/2, n >= 1
The left inequality holds if c1 = 1/14, n >= 7
So we let c1 = 1/14, c2 = 1/2, and n0 = 7. 

                                         3.1.4
 We can also verify that 6*n^3 != Theta(n^2)
Suppose 6*n^3 = Theta(n^2), then in particular
there are c2 and n0 such that 6*n^3 <= c2*n^2
for n >= n0.  But this means that n <= c2/6
for n >= n0 (after dividing by n^2), which
can't happen for large n since c2 is constant.

 Intuitively, even a tiny fraction of the
highest-order term is enough to dominate the
lower-order terms for large n.  Thus setting
c1 to a value slightly smaller than the 
coefficient of the higher-order term, and c2
to a value slightly larger will work in the
definition of Theta.  Also, we can ignore the
coefficient of the higher-order term, since
changing it only changes c1 and c2 by the
same constant factor. 

 Any polynomial p(n) = Theta(n^d) where

            d
  p(n) = Sum ( a_i * n^i )
          i=0

and a_d > 0 (Problem 3-1 c.)

 The special case d = 2 is shown on page 44.

 Also, any constant, c, is a 0-degree
polynomial so c = Theta(n^0) = Theta(1)

O-notation                               3.1.5

 Theta notation bounds a function both above
and below.  If we only have an asymptotic
upper bound, we use O-notation.  

 O(g(n)) (pronounced "big-oh of g of n" or
just "oh of g of n") is defined as:

O(g(n)) = { f : there exist positive constants
                c and n0 such that
                0 <= f(n) <= cg(n)
                for all n >= n0 }

 We use O-notation to give an upper bound on a
function.  Figure 3.1(b) (page 43) shows an
example: for all n to the right of n0, f(n) is
on or below cg(n).

 We write f(n) = O(g(n)) to indicate that f is
a member of the set O(g(n)).

Notes: f(n) = Theta(g(n)) ==> f(n) = O(g(n)),
     i.e. Theta(g(n)) is contained in O(g(n)).
 
 - Any linear function a*n + b is in O(n^2)

 - WARNING!  In some books O is used to mean
  what we call Theta.

 - Since O describes an upper bound, it is
  used to describe worst-case running-time.

Omega-notation                           3.1.6

 Just as O provides an asymptotic upper bound,
Omega-notation provides an asymptotic lower
bound.  Omega(g(n)) (pronounced "big-omega of
g of n" or just "omega of g of n") is defined:

Omega(g(n)) = { f : there exist positive
                    constants c and n0 such
                    that 0 <= cg(n) <= f(n)
                    for all n >= n0 }

 We use Omega-notation to give a lower bound
on a function.  Figure 3.1(c) (page 43) shows
an example: for all n to the right of n0,
f(n) is on or above cg(n).

 We write f(n) = Omega(g(n)) to indicate that
f is a member of the set Omega(g(n)).

Theorem 3.1: For any two functions f and g,
f(n) = Theta(g(n)) if and only if 
f(n) = O(g(n)) and f(n) = Omega(g(n))

 - Since Omega describes a lower bound, it is
  used to describe best-case running-time.

 - The running time of insertion sort falls
  between Omega(n) (already sorted) and O(n^2)
  (reverse sorted).  For insertion sort it is
  Theta(n) for sorted input and Theta(n^2) for
  reverse-sorted (or even average) input.

Asymptotic notation in equations and     3.1.7
                                  inequalities

Question: How do we interpret a formula like:

   2n^2 + 3n + 1 = 2n^2 + Theta(n)

Answer: Theta(n) stands for some function that
 we don't want to name, but that makes the
 equation true: 2n^2 + 3n + 1 = 2n^2 + f(n)
 where f(n) is in Theta(n) (now f(n) = 3n + 1
 here, and 3n + 1 is indeed in Theta(n) ).

 As another example, the running time of merge
sort is given by the recurrence:

   T(n) = 2T(n/2) + Theta(n)

 Asymptotic notation can occur on both sides
of an equation, as in:

   2n^2 + Theta(n) = Theta(n^2)

 What this means is that for any function in
Theta(n), there is a way to choose a function
in Theta(n^2) so that the equation is true.

 These relationships can be chained together:

   2n^2 + 3n + 1 = 2n^2 + Theta(n)
                 = Theta(n^2)

                                         3.1.8
 The first equation says that there is _some_
function f(n) in Theta(n) that makes it true
for all n.  The second equation says that for
_any_ function g(n) in Theta(n) (such as f(n))
there is _some_ function h(n) in Theta(n^2)
such that 2n^2 + g(n) = h(n) for all n.  This
means that 2n^2 + 3n + 1 = Theta(n^2), which
is what we see intuitively from the equations.

o-notation

 2n^2 = O(n^2) is asymptotically tight, but
the bound 2n = O(n^2) is not.  We use
o-notation to denote an upper bound that is
not asymptotically tight.  We define o(g(n))
(read "little-oh of g of n") as the set:

o(g(n)) = { f : for any positive constant c
                there is a constant n0 > 0
                such that 0 <= f(n) < cg(n)
                for all n >= n0 }

 E.g. 2n = o(n^2) but 2n^2 is not in o(n^2)

 Intuitively, in the o-notation, f(n) becomes
insignificant relative to g(n) as n goes to
infinity:
          lim f(n)/g(n) = 0
         n-->inf

omega-notation                           3.1.9

 We use omega-notation to denote a lower bound
that is not asymptotically tight.  So:
omega-notation is to Omega-notation as
o-notation is to O-notation.  We define
omega(g(n)) (read "little-omega of g of n") as

omega(g(n)) = { f : for any constant c > 0,
                there is a constant n0 > 0
                such that 0 <= cg(n) < f(n)
                for all n >= n0 }

 2n^2 = omega(n) but 2n^2 is not in omega(n^2)

 Intuitively, in omega-notation, f(n) becomes
arbitrarily large relative to g(n) as n goes
to infinity:
             lim f(n)/g(n) = infinity
            n-->inf
if the limit exists.

Comparison of functions

 Some relational properties that apply to real
numbers apply to asymptotic comparisons also.

Reflexivity:
 f(n) = Theta(f(n))
 f(n) = O(f(n))
 f(n) = Omega(f(n))

Transitivity:                           3.1.10
 f(n) = Theta(g(n)) and g(n) = Theta(h(n))
                        ==> f(n) = Theta(h(n))
 f(n) = O(g(n)) and g(n) = O(h(n))
                            ==> f(n) = O(h(n))
 f(n) = Omega(g(n)) and g(n) = Omega(h(n))
                        ==> f(n) = Omega(h(n))
 f(n) = o(g(n)) and g(n) = o(h(n))
                            ==> f(n) = o(h(n))
 f(n) = omega(g(n)) and g(n) = omega(h(n))
                        ==> f(n) = omega(h(n))
Symmetry:
 f(n) = Theta(g(n)) <==> g(n) = Theta(f(n))

Transpose symmetry:
 f(n) = O(g(n)) <==> g(n) = Omega(f(n))
 f(n) = o(g(n)) <==> g(n) = omega(f(n))

Analogy with real number comparisons:
 f(n) = Theta(g(n)) is like a = b
 f(n) = O(g(n))     is like a <= b
 f(n) = Omega(g(n)) is like a >= b
 f(n) = o(g(n))     is like a < b
 f(n) = omega(g(n)) is like a > b

We say: f(n) is "asymptotically smaller" than
       g(n) if f(n) = o(g(n)) and
We say: f(n) is "asymptotically larger" than
       g(n) if f(n) = omega(g(n))

 All reals can be compared, but not functions:
for f(n) = n and g(n) = n^( 1 + sin(n) )
neither f(n) = O(g(n)) nor f(n) = Omega(g(n))

3.2 Standard notations and common        3.2.1
                                  functions 
Monotonicity

 A function f is monotonically increasing if
m <= n implies f(m) <= f(n).
 A function f is monotonically decreasing if
m <= n implies f(m) >= f(n).
 A function f is strictly increasing if m < n
implies f(m) < f(n).
 A function f is strictly decreasing if m < n
implies f(m) > f(n).


Floors and Ceilings

 For any real number x, floor(x) is the
greatest integer <= x, and ceiling(x) is the
smallest integer >= x.  Both functions are
monotonically increasing.  For any real x:
 x - 1 < floor(x) <= x <= ceiling(x) < x + 1

 For any integer n,
    floor(n/2) + ceiling(n/2) = n

 For any real n >= 0 and integers a, b > 0
ceiling( ceiling(n/a)/b ) = ceiling( n/(ab) )
floor( floor(n/a)/b ) = floor( n/(ab) )
ceiling( a/b ) <= (a + (b - 1))/b
floor( a/b ) >= (a - (b - 1))/b

Modular arithmetic                       3.2.2

 For any integer a and any positive integer n,
the value of  a mod n  is the remainder (or
residue) of the quotient a/n:

   a mod n = a - floor(a/n)*n

 To express equality of remainders, we use the
"equivalence" symbol: three horizontal lines;
we use the word "equiv" in these notes for it.
I.e. if (a mod n) = (b mod n), we write
"a equiv b (mod n)" and say "a is equivalent
to b, modulo n".  So  a equiv b (mod n)  <==>
a - b is divisible by n.  We also write
"a not equiv b (mod n)" if this is not so.


Polynomials

 Given a nonnegative integer d, a polynomial
in n of degree d is a function of the form:
            d
  p(n) = Sum ( a_i * n^i )
          i=0
where the constant a_i are the coefficients of
the polynomial and a_d is not 0.
 A polynomial is asymptotically positive if
a_d > 0, and then p(n) = Theta(n^d).  For any
real constant a >= 0, n^a is monotonically
increasing; if a <= 0, n^a is monotonically
decreasing.  A function f is polynomially
bounded if f(n) = O(n^k) for some constant k.

Exponentials                             3.2.3

 For all real a > 0, m, and n, we have:

   a^0 = 1  a^1 = a  a^(-1) = 1/a

   (a^m)^n = a^(mn) = (a^n)^m

   (a^m)*(a^n) = a^(m+n)

 For all n and a >= 1, the function a^n is
monotonically increasing in n, and we assume
that 0^0 = 1 when convenient.

 For all real constants a > 1 and b:
    lim n^b / a^n = 0                 (3.9)
  n-->inf
from which we conclude:   n^b = o(a^n).

 Using e = 2.71828... to denote the base of
the natural logarithms, then for all real x:
                              inf
e^x = 1 + x + x^2/2! + ... = Sum (x^i/i!)
                            i = 0
 For for all real x:  e^x >= 1 + x, where "="
holds only when x = 0.  When |x| <= 1, we have
1 + x <= e^x <= 1 + x + x^2, so when x --> 0,
1 + x is a very good approximation.

 For all x:
    lim (1 + x/n)^n = e^x
  n-->inf

Logarithms                               3.2.4

 We use the following notations:
   lg n = log_2(n)    (binary logarithm)
   ln n = log_e(n)    (natural logarithm)
 lg^k n = (lg n)^k    (exponentiation)
lg lg n = lg(lg(n))   (composition of logs)

 Important notational convention: logarithm
functions will apply only to the next term in
the formula, so that lg n + k means (lg n) + k
and not lg(n + k).
 If b > 1 is constant, then log_b(n) is a
strictly increasing function of n (for n > 0).

 For all real a > 0, b > 0, c > 0, and n:

         a = b^(log_b a)
 log_c(ab) = log_c(a) + log_c(b)
log_b(a^n) = n log_b(a)
log_b(1/a) = -log_b(a)
   log_b a = log_c(a)/log_c(b)   (3.14)
   log_b a = 1/log_a(b)
a^log_b(c) = b^(log_b(a)*log_b(c))
           = c^log_b(a) 

 By (3.14), changing the base of a logarithm
(from b to c), changes the logarithm by a
constant factor ( 1/log_c(b) ) which we don't
care about in asymptotic notation, so we
usually use lg for logarithms, since 2 is the
most natural base when analyzing algorithms.

 There is a simple series expansion      3.2.5
for ln(1 + x) when |x| < 1 :

ln(1 + x) = x - x^2/2 + x^3/3 - x^4/4 + ...

 The following inequalities hold for x > -1 :

x/(1 + x) <= ln(1 + x) <= x
with equality only for x = 0.

 A function f is polylogarithmically bounded
if f(n) = O( lg^k n ) for some constant k.  By
substituting lg n for n and 2^a for a in (3.9)

  lim lg^b(n)/(2^a)^lg(n) =  lim lg^b(n)/n^a
n-->inf                    n-->inf
                      = 0
for a > 0, from which we get lg^b(n) = o(n^a),
so any positive polynomial function grows
faster than any polylogarithmic function.


Factorials

 n! is defined for integers n >= 0 as:
     n! = 1         if n = 0
          n*(n-1)!  if n > 0
i.e. n! = 1 * 2 * 3 * ... * n

 A weak upper bound on n! is n^n, since each
factor in the product is <= n.  Stirling's
approximation gives a tighter bound:

n! = sqrt(2pi*n) * (n/e)^n * (1 + Theta(1/n))

 One can prove (Exercise 3.2-3):         3.2.6

    n! = o(n^n)
    n! = omega(2^n)
lg(n!) = Theta(n*lg(n))

 The following equation also holds for n >= 1:

n! = sqrt(2pi*n) * (n/e)^n * e^(alpha_n)

where  1/(12n + 1) < alpha_n < 1/(12n)


Functional notation

 For and integer i >= 0 we define the function
iteratively applied i times by:
  (i)
 f  (n) = n            if i = 0
             (i-1) 
          f(f    (n))  if i > 0

                            (i)
 For example if f(n) = 2n, f  (n) = 2^i * n.

The iterated logarithm function          3.2.7

 We use lg*(n) to denote the iterated
                                     (i)
logarithm, defined as follows: let lg  (n) be
defined as above, with f(n) = lg(n).  Because
lg is only defined for positive numbers, we
       (i-1)
need lg    (n) > 0.  Then lg* is defined as:
                              (i)
     lg*(n) = min{ i >= 0 : lg  (n) <= 1 }

 The iterated logarithm grows _very_ slowly:

  lg*(2) = 1  since lg(2) = 1
  lg*(2^2) = lg*(4) = 2, since lg(lg(4)) = 1
  lg*(2^4) = lg*(16) = 3
  lg*(2^16) = lg*(65536) = 4
  lg*(2^65536) = 5

Since 2^65536 is about 10^20000, it is very
much bigger than 10^80, the number of atoms in
the observable universe, so we rarely see an
input size n such that lg*(n) > 5.


Fibonacci numbers

 The Fibonacci numbers are defined by:

F_0 = 0,
F_1 = 1,
F_i = F_(i-1) + F_(i-2)  for i >= 2

 The first few are:                      3.2.8
0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, ...

 They are related to the golden ratio phi and
to its conjugate, phi-hat, which are given by:

 phi     = (1 + sqrt(5))/2 = 1.618034...

 phi-hat = (1 - sqrt(5))/2 = -.618034...

Specifically:

 F_i = ( phi^i - phi-hat^i ) / sqrt(5)

which can be proved by induction (Exercise
3.2-6 page 57).  Since |phi-hat| < 1, we have:

|phi-hat^i|/sqrt(5) < 1/sqrt(5) < 1/2, so that

F_i is equal to phi^i/sqrt(5) rounded to the

nearest integer.  Thus Fibonacci numbers grow
exponentially.