Chapter 3 Growth of Functions 3.1.1 "Asymptotic" notation is used to answer the question: What is the LIMIT of an algorithm's running time when the size of its input increases WITHOUT BOUND? This is the algorithm's asymptotic efficiency 3.1 Asymptotic notation The functions we will use to measure asymptotic efficiency are defined on the set of natural numbers N = {0,1,2,...} -- for example the worst-case running-time function T(n). Occasionally it will be convenient to abuse the notation by extending it to real numbers or restricting it to a subset of N (the even numbers, for example). But we must be careful not to mis-use the notation. Theta-notation 3.1.2 We found that the worst-case running time of insertion sort is T(n) = Theta(n^2). Here is the precise definition of that notation. Definition: For a given function g(n), we denote by Theta(g(n)) the set of functions: Theta(g(n)) = { f : there exist positive constants c1, c2, and n0 such that 0 <= c1g(n) <= f(n) <= c2g(n) for all n >= n0 } So a function f belongs to Theta(g(n)) if there exist positive constants c1 and c2 such that f(n) can be "sandwiched" between c1g(n) and c2g(n) for sufficiently large n. Note that we should say that "f is a member of Theta(g(n))", but it is almost always expressed as "f(n) = Theta(g(n))", which may be a bit confusing, but has advantages. Figure 3.1(a) shows this situation: for all n to the right of n0, the value of f(n) lies at or above c1g(n) and at or below c2g(n). In this case, we say g(n) is an "asymptotically tight" bound for f(n). 3.1.3 The definition of Theta(g(n)) requires that each member function f of it be asymptotically nonnegative, i.e. f(n) >= 0 whenever n is sufficiently large. (Similarly we say a function f is asymptotically positive if f(n) > 0 whenever n is sufficiently large.) Consequently g(n) must be asymptotically nonnegative, or else Theta(g(n)) is empty. We therefore assume that all functions used in the Theta notation and other asymptotic notations are asymptotically nonnegative. Previously we used an informal notation for Theta that involved ignoring lower-order terms and the coefficient of the highest-order term. We justify this technique for a specific case -- we show that: 1/2*n^2 - 3n = Theta(n^2) We must find positive constants c1, c2, and n0 such that: c1*n^2 <= 1/2*n^2 - 3n <= c2*n^2 for all n >= n0. Dividing by n^2 gives: c1 <= 1/2 - 3/n <= c2 The right inequality holds if c2 = 1/2, n >= 1 The left inequality holds if c1 = 1/14, n >= 7 So we let c1 = 1/14, c2 = 1/2, and n0 = 7. 3.1.4 We can also verify that 6*n^3 != Theta(n^2) Suppose 6*n^3 = Theta(n^2), then in particular there are c2 and n0 such that 6*n^3 <= c2*n^2 for n >= n0. But this means that n <= c2/6 for n >= n0 (after dividing by n^2), which can't happen for large n since c2 is constant. Intuitively, even a tiny fraction of the highest-order term is enough to dominate the lower-order terms for large n. Thus setting c1 to a value slightly smaller than the coefficient of the higher-order term, and c2 to a value slightly larger will work in the definition of Theta. Also, we can ignore the coefficient of the higher-order term, since changing it only changes c1 and c2 by the same constant factor. Any polynomial p(n) = Theta(n^d) where d p(n) = Sum ( a_i * n^i ) i=0 and a_d > 0 (Problem 3-1 c.) The special case d = 2 is shown on page 44. Also, any constant, c, is a 0-degree polynomial so c = Theta(n^0) = Theta(1) O-notation 3.1.5 Theta notation bounds a function both above and below. If we only have an asymptotic upper bound, we use O-notation. O(g(n)) (pronounced "big-oh of g of n" or just "oh of g of n") is defined as: O(g(n)) = { f : there exist positive constants c and n0 such that 0 <= f(n) <= cg(n) for all n >= n0 } We use O-notation to give an upper bound on a function. Figure 3.1(b) (page 43) shows an example: for all n to the right of n0, f(n) is on or below cg(n). We write f(n) = O(g(n)) to indicate that f is a member of the set O(g(n)). Notes: f(n) = Theta(g(n)) ==> f(n) = O(g(n)), i.e. Theta(g(n)) is contained in O(g(n)). - Any linear function a*n + b is in O(n^2) - WARNING! In some books O is used to mean what we call Theta. - Since O describes an upper bound, it is used to describe worst-case running-time. Omega-notation 3.1.6 Just as O provides an asymptotic upper bound, Omega-notation provides an asymptotic lower bound. Omega(g(n)) (pronounced "big-omega of g of n" or just "omega of g of n") is defined: Omega(g(n)) = { f : there exist positive constants c and n0 such that 0 <= cg(n) <= f(n) for all n >= n0 } We use Omega-notation to give a lower bound on a function. Figure 3.1(c) (page 43) shows an example: for all n to the right of n0, f(n) is on or above cg(n). We write f(n) = Omega(g(n)) to indicate that f is a member of the set Omega(g(n)). Theorem 3.1: For any two functions f and g, f(n) = Theta(g(n)) if and only if f(n) = O(g(n)) and f(n) = Omega(g(n)) - Since Omega describes a lower bound, it is used to describe best-case running-time. - The running time of insertion sort falls between Omega(n) (already sorted) and O(n^2) (reverse sorted). For insertion sort it is Theta(n) for sorted input and Theta(n^2) for reverse-sorted (or even average) input. Asymptotic notation in equations and 3.1.7 inequalities Question: How do we interpret a formula like: 2n^2 + 3n + 1 = 2n^2 + Theta(n) Answer: Theta(n) stands for some function that we don't want to name, but that makes the equation true: 2n^2 + 3n + 1 = 2n^2 + f(n) where f(n) is in Theta(n) (now f(n) = 3n + 1 here, and 3n + 1 is indeed in Theta(n) ). As another example, the running time of merge sort is given by the recurrence: T(n) = 2T(n/2) + Theta(n) Asymptotic notation can occur on both sides of an equation, as in: 2n^2 + Theta(n) = Theta(n^2) What this means is that for any function in Theta(n), there is a way to choose a function in Theta(n^2) so that the equation is true. These relationships can be chained together: 2n^2 + 3n + 1 = 2n^2 + Theta(n) = Theta(n^2) 3.1.8 The first equation says that there is _some_ function f(n) in Theta(n) that makes it true for all n. The second equation says that for _any_ function g(n) in Theta(n) (such as f(n)) there is _some_ function h(n) in Theta(n^2) such that 2n^2 + g(n) = h(n) for all n. This means that 2n^2 + 3n + 1 = Theta(n^2), which is what we see intuitively from the equations. o-notation 2n^2 = O(n^2) is asymptotically tight, but the bound 2n = O(n^2) is not. We use o-notation to denote an upper bound that is not asymptotically tight. We define o(g(n)) (read "little-oh of g of n") as the set: o(g(n)) = { f : for any positive constant c there is a constant n0 > 0 such that 0 <= f(n) < cg(n) for all n >= n0 } E.g. 2n = o(n^2) but 2n^2 is not in o(n^2) Intuitively, in the o-notation, f(n) becomes insignificant relative to g(n) as n goes to infinity: lim f(n)/g(n) = 0 n-->inf omega-notation 3.1.9 We use omega-notation to denote a lower bound that is not asymptotically tight. So: omega-notation is to Omega-notation as o-notation is to O-notation. We define omega(g(n)) (read "little-omega of g of n") as omega(g(n)) = { f : for any constant c > 0, there is a constant n0 > 0 such that 0 <= cg(n) < f(n) for all n >= n0 } 2n^2 = omega(n) but 2n^2 is not in omega(n^2) Intuitively, in omega-notation, f(n) becomes arbitrarily large relative to g(n) as n goes to infinity: lim f(n)/g(n) = infinity n-->inf if the limit exists. Comparison of functions Some relational properties that apply to real numbers apply to asymptotic comparisons also. Reflexivity: f(n) = Theta(f(n)) f(n) = O(f(n)) f(n) = Omega(f(n)) Transitivity: 3.1.10 f(n) = Theta(g(n)) and g(n) = Theta(h(n)) ==> f(n) = Theta(h(n)) f(n) = O(g(n)) and g(n) = O(h(n)) ==> f(n) = O(h(n)) f(n) = Omega(g(n)) and g(n) = Omega(h(n)) ==> f(n) = Omega(h(n)) f(n) = o(g(n)) and g(n) = o(h(n)) ==> f(n) = o(h(n)) f(n) = omega(g(n)) and g(n) = omega(h(n)) ==> f(n) = omega(h(n)) Symmetry: f(n) = Theta(g(n)) <==> g(n) = Theta(f(n)) Transpose symmetry: f(n) = O(g(n)) <==> g(n) = Omega(f(n)) f(n) = o(g(n)) <==> g(n) = omega(f(n)) Analogy with real number comparisons: f(n) = Theta(g(n)) is like a = b f(n) = O(g(n)) is like a <= b f(n) = Omega(g(n)) is like a >= b f(n) = o(g(n)) is like a < b f(n) = omega(g(n)) is like a > b We say: f(n) is "asymptotically smaller" than g(n) if f(n) = o(g(n)) and We say: f(n) is "asymptotically larger" than g(n) if f(n) = omega(g(n)) All reals can be compared, but not functions: for f(n) = n and g(n) = n^( 1 + sin(n) ) neither f(n) = O(g(n)) nor f(n) = Omega(g(n)) 3.2 Standard notations and common 3.2.1 functions Monotonicity A function f is monotonically increasing if m <= n implies f(m) <= f(n). A function f is monotonically decreasing if m <= n implies f(m) >= f(n). A function f is strictly increasing if m < n implies f(m) < f(n). A function f is strictly decreasing if m < n implies f(m) > f(n). Floors and Ceilings For any real number x, floor(x) is the greatest integer <= x, and ceiling(x) is the smallest integer >= x. Both functions are monotonically increasing. For any real x: x - 1 < floor(x) <= x <= ceiling(x) < x + 1 For any integer n, floor(n/2) + ceiling(n/2) = n For any real n >= 0 and integers a, b > 0 ceiling( ceiling(n/a)/b ) = ceiling( n/(ab) ) floor( floor(n/a)/b ) = floor( n/(ab) ) ceiling( a/b ) <= (a + (b - 1))/b floor( a/b ) >= (a - (b - 1))/b Modular arithmetic 3.2.2 For any integer a and any positive integer n, the value of a mod n is the remainder (or residue) of the quotient a/n: a mod n = a - floor(a/n)*n To express equality of remainders, we use the "equivalence" symbol: three horizontal lines; we use the word "equiv" in these notes for it. I.e. if (a mod n) = (b mod n), we write "a equiv b (mod n)" and say "a is equivalent to b, modulo n". So a equiv b (mod n) <==> a - b is divisible by n. We also write "a not equiv b (mod n)" if this is not so. Polynomials Given a nonnegative integer d, a polynomial in n of degree d is a function of the form: d p(n) = Sum ( a_i * n^i ) i=0 where the constant a_i are the coefficients of the polynomial and a_d is not 0. A polynomial is asymptotically positive if a_d > 0, and then p(n) = Theta(n^d). For any real constant a >= 0, n^a is monotonically increasing; if a <= 0, n^a is monotonically decreasing. A function f is polynomially bounded if f(n) = O(n^k) for some constant k. Exponentials 3.2.3 For all real a > 0, m, and n, we have: a^0 = 1 a^1 = a a^(-1) = 1/a (a^m)^n = a^(mn) = (a^n)^m (a^m)*(a^n) = a^(m+n) For all n and a >= 1, the function a^n is monotonically increasing in n, and we assume that 0^0 = 1 when convenient. For all real constants a > 1 and b: lim n^b / a^n = 0 (3.9) n-->inf from which we conclude: n^b = o(a^n). Using e = 2.71828... to denote the base of the natural logarithms, then for all real x: inf e^x = 1 + x + x^2/2! + ... = Sum (x^i/i!) i = 0 For for all real x: e^x >= 1 + x, where "=" holds only when x = 0. When |x| <= 1, we have 1 + x <= e^x <= 1 + x + x^2, so when x --> 0, 1 + x is a very good approximation. For all x: lim (1 + x/n)^n = e^x n-->inf Logarithms 3.2.4 We use the following notations: lg n = log_2(n) (binary logarithm) ln n = log_e(n) (natural logarithm) lg^k n = (lg n)^k (exponentiation) lg lg n = lg(lg(n)) (composition of logs) Important notational convention: logarithm functions will apply only to the next term in the formula, so that lg n + k means (lg n) + k and not lg(n + k). If b > 1 is constant, then log_b(n) is a strictly increasing function of n (for n > 0). For all real a > 0, b > 0, c > 0, and n: a = b^(log_b a) log_c(ab) = log_c(a) + log_c(b) log_b(a^n) = n log_b(a) log_b(1/a) = -log_b(a) log_b a = log_c(a)/log_c(b) (3.14) log_b a = 1/log_a(b) a^log_b(c) = b^(log_b(a)*log_b(c)) = c^log_b(a) By (3.14), changing the base of a logarithm (from b to c), changes the logarithm by a constant factor ( 1/log_c(b) ) which we don't care about in asymptotic notation, so we usually use lg for logarithms, since 2 is the most natural base when analyzing algorithms. There is a simple series expansion 3.2.5 for ln(1 + x) when |x| < 1 : ln(1 + x) = x - x^2/2 + x^3/3 - x^4/4 + ... The following inequalities hold for x > -1 : x/(1 + x) <= ln(1 + x) <= x with equality only for x = 0. A function f is polylogarithmically bounded if f(n) = O( lg^k n ) for some constant k. By substituting lg n for n and 2^a for a in (3.9) lim lg^b(n)/(2^a)^lg(n) = lim lg^b(n)/n^a n-->inf n-->inf = 0 for a > 0, from which we get lg^b(n) = o(n^a), so any positive polynomial function grows faster than any polylogarithmic function. Factorials n! is defined for integers n >= 0 as: n! = 1 if n = 0 n*(n-1)! if n > 0 i.e. n! = 1 * 2 * 3 * ... * n A weak upper bound on n! is n^n, since each factor in the product is <= n. Stirling's approximation gives a tighter bound: n! = sqrt(2pi*n) * (n/e)^n * (1 + Theta(1/n)) One can prove (Exercise 3.2-3): 3.2.6 n! = o(n^n) n! = omega(2^n) lg(n!) = Theta(n*lg(n)) The following equation also holds for n >= 1: n! = sqrt(2pi*n) * (n/e)^n * e^(alpha_n) where 1/(12n + 1) < alpha_n < 1/(12n) Functional notation For and integer i >= 0 we define the function iteratively applied i times by: (i) f (n) = n if i = 0 (i-1) f(f (n)) if i > 0 (i) For example if f(n) = 2n, f (n) = 2^i * n. The iterated logarithm function 3.2.7 We use lg*(n) to denote the iterated (i) logarithm, defined as follows: let lg (n) be defined as above, with f(n) = lg(n). Because lg is only defined for positive numbers, we (i-1) need lg (n) > 0. Then lg* is defined as: (i) lg*(n) = min{ i >= 0 : lg (n) <= 1 } The iterated logarithm grows _very_ slowly: lg*(2) = 1 since lg(2) = 1 lg*(2^2) = lg*(4) = 2, since lg(lg(4)) = 1 lg*(2^4) = lg*(16) = 3 lg*(2^16) = lg*(65536) = 4 lg*(2^65536) = 5 Since 2^65536 is about 10^20000, it is very much bigger than 10^80, the number of atoms in the observable universe, so we rarely see an input size n such that lg*(n) > 5. Fibonacci numbers The Fibonacci numbers are defined by: F_0 = 0, F_1 = 1, F_i = F_(i-1) + F_(i-2) for i >= 2 The first few are: 3.2.8 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, ... They are related to the golden ratio phi and to its conjugate, phi-hat, which are given by: phi = (1 + sqrt(5))/2 = 1.618034... phi-hat = (1 - sqrt(5))/2 = -.618034... Specifically: F_i = ( phi^i - phi-hat^i ) / sqrt(5) which can be proved by induction (Exercise 3.2-6 page 57). Since |phi-hat| < 1, we have: |phi-hat^i|/sqrt(5) < 1/sqrt(5) < 1/2, so that F_i is equal to phi^i/sqrt(5) rounded to the nearest integer. Thus Fibonacci numbers grow exponentially.