Chapter 4 Recurrences 4.0.1
The running time of a divide-and-conquer
algorithm can often be described by a
Recurrence: an equation or inequality that
describes a function in terms of its value
on smaller inputs.
For example, the running time of merge-sort:
T(n) = / Theta(1) if n = 1
\ 2T(n/2) + Theta(n) if n > 1
Three methods for solving recurrences -- i.e.
obtaining Theta or big-O bounds on T(n):
1) The substitution method: make a guess and
then prove the guess is correct by induction.
2) The recursion-tree method: convert the
recurrence into a tree of costs & sum them.
3) The "master method": gives bounds for
recurrences of the form: T(n) = aT(n/b)+f(n)
where a >= 1, b > 1, and f(n) is given.
Technicalities 4.0.2
1. Usually the running time T(n) is only
defined for integer n. This is often ignored
in practice. For instance, the precise
recurrence for the run time of merge sort is:
T(n) = / Theta(1) if n = 1
\ T(floor(n/2)) + T(ceiling(n/2)) +
Theta(n) if n > 1
2. We usually don't give boundary conditions,
since T(n) = Theta(1) for small n. Usually
changing T(1) only changes the solution to
the recurrence by a constant factor. So, for
example, we usually just state the recurrence
for the run time of merge sort simply as:
T(n) = 2T(n/2) + Theta(n)
Thus we usually omit floors, ceilings, and
boundary conditions when we solve recurrences.
Usually they don't matter, but we check to
make sure that that is the case after we have
obtained our solution.
4.1 The substitution method 4.1.1
The substitution method involves two steps:
1. Guess the form of the solution.
2. Use mathematical induction to find the
constants and show that the solution works.
This method can establish upper or lower
bounds or both on a recurrence. For example,
we determine an upper bound on the recurrence:
T(n) = 2T(floor(n/2)) + n (4.4)
1. We guess the solution is T(n) = O(n lg n),
so we need to prove 2. T(n) <= cn lg(n) for an
appropriate choice of c. We start by assuming
this holds for floor(n/2), i.e. that
T(floor(n/2)) <= cfloor(n/2) lg(floor(n/2))
Substituting this into (4.4) above gives:
T(n) <= 2[cfloor(n/2) lg(floor(n/2))] + n
<= cn lg(n/2) + n
= cn lg(n) - cn lg(2) + n
= cn lg(n) - cn + n
<= cn lg(n)
where the last step holds if c >= 1.
To complete the inductive proof, we need to
pick c so that the inequality holds for the
boundary case too. Now T(1) > 0, but lg(1) is
0, which seems to be a problem.
4.1.2
But we only need T(n) <= cn lg(n) for n >= n0,
for some n0 > 0. We can let n0 = 2 and pick c
so that T(2) <= c 2 lg(2) and T(3) < c 3 lg(3)
to establish the base cases n = 2 and n = 3.
Since T(2) = 2T(floor(2/2)) + 2 = 2T(1) + 2,
and T(3) = 2T(floor(3/2)) + 3 = 2T(1) + 3,
then letting c = maximum of (2T(1)+2)/(2lg2)
and (2T(1)+3)/(3lg3) will work. The text
shows that if T(1) = 1, then c = 2.
Making a good guess
Of course guessing is an art. But here are
two methods:
1. If a recurrence is similar to one you have
solved before, the solution to the new
recurrence will be similar to the old one.
Consider for example the recurrence:
T(n) = 2T(floor(n/2) + 17) + n
which is similar to (4.4) above. So you might
correctly guess that T(n) = O(n lg n), which
can be proved by the substitution method.
2. Guess easy upper and lower bounds, such as
T(n) = Omega(n) and T(n) = O(n^2) for (4.4),
and then improve them until they converge to
the asymptotically tight T(n) = Theta(n lg n).
Subtleties 4.1.3
A trick may be needed to make the solution go
through -- subtracting a lower order term for
example. Consider the recurrence:
T(n) = T(floor(n/2)) + T(ceiling(n/2)) + 1
It is natural to guess the solution is O(n).
But trying to prove T(n) <= cn leads to:
T(n) <= c floor(n/2) + c ceiling(n/2) + 1
= c n + 1
which does not give T(n) <= c n for any choice
of c. Now T(n) = O(n) is correct, but to show
it, we have to guess that T(n) <= c n - b, so:
T(n) <= c floor(n/2) - b + c ceiling(n/2) - b
+ 1
= c n - 2 b + 1
<= c n - b
if we let b >= 1. As before, we must also
choose c to handle the boundary conditions.
Avoiding pitfalls
It is easy to make mistakes. For example, we
can "prove" T(n) = O(n) for (4.4) by guessing
T(n) <= cn and then arguing:
T(n) <= 2 c floor(n/2) + n
<= c n + n
= O(n) <=== Wrong!!
The error is that we haven't proved the exact
form of the inductive hypothesis: T(n) <= cn.
Changing variables 4.1.4
Sometimes algebraic manipulation can make an
unknown recurrence look like one you have
solved. For example, consider the recurrence:
T(n) = 2T(floor(sqrt(n))) + lg n
Ignoring round-to-integer problems, we let
m = lg(n), so that n = 2^m, and we get:
T(2^m) = 2T(2^(m/2)) + m
And now, letting S(m) = T(2^m), we have:
S(m) = 2S(m/2) + m
which we suspect (and can show) has the
solution S(m) = O(m lg m). Changing back to
T(n), T(n) = T(2^m) = S(m) = O(m lg m) =
O(lg n lg lg n).
4.2 The recursion-tree method 4.2.1
In a recursion tree, each node represents the
cost of a recursive call of the algorithm
somewhere in its recursive invocation, but
ignoring the cost of further recursive calls
from within it, which are accounted for by
"child" costs of that node. Once the costs of
each node have been determined, the costs of
the nodes at each level are summed, and then
the per-level costs are summed to get the
final answer. Recursion trees are especially
useful in finding the running time of a
divide-and-conquer algorithm
Recursion trees are often used to generate
good guesses, which can then be verified by
the substitution method. Used in this way,
we can tolerate a bit of "sloppiness" in the
calculations. Or, by being careful, we can
turn the guess into a proof (which is done in
Section 4.4 when proving the master method).
We first use a recursion tree to guess a
solution to the recurrence:
T(n) = 3T(floor(n/4)) + Theta(n^2)
We want to find an upper bound for T(n). We
ignore the floor function and non-integer
values of n/4, so we can change the recurrence
to: T(n) = 3T(n/4) + cn^2 for some c > 0.
Figure 4.1, page 69, shows the construction of
the recursion tree for this recurrence.
4.2.2
Figure 4.1(b) shows one expansion of T(n)
into the non-recursive cost, cn^2, of the root
and the three recursive costs, T(n/4), of its
children:
cn^2
/ | \
/ | \
/ | \
/ | \
/ | \
T(n/4) T(n/4) T(n/4)
Figure 4.1(c) shows the expansion of T(n) one
level deeper, with each of the T(n/4) costs
replaced by the non-recursive cost, c(n/4)^2,
and the three recursive costs, T(n/16), of its
children. We continue this process until we
reach the boundary condition, a subproblem of
size 1. If this occurs at depth i, n/4^i = 1,
i.e. i = log_4(n). Thus the tree has
log_4(n) + 1 levels: 0, 1, 2, ..., log_4(n),
as is shown in Figure 4.1(d).
To get the per-level cost, we note that the
non-recursive cost at each level decreases by
a factor of 4 and there are 3 times as many of
them as in the previous level. So node cost
at level i is c(n/4^i)^2 and there are 3^i
such costs for a total of (3/16)^i cn^2. At
the last level of depth log_4(n), there are
3^log_4(n) = n^log_4(3) nodes of cost T(1) for
a total cost of Theta(n^log_4(3)).
Thus the total cost of all levels is: 4.2.3
log_4(n)-1
T(n) = Sum ( (3/16)^i cn^2 ) +
i=0 Theta(n^log_4(3))
inf
< Sum( (3/16)^i cn^2) + Theta(n^log_4(3))
i=0
= 1/(1 - 3/16)cn^2 + Theta(n^log_4(3))
= 16/13 cn^2 + Theta(n^log_4(3))
= O(n^2)
Which gives us our guess. Note that 13/13 of
the 16/13 of the cn^2 term comes from the root
and so T(n) = Omega(n^2), since it must take
at least as much time to run as for the root.
We now use the substitution method to verify
our guess, T(n) = O(n^2) as an upper bound for
the original recurrence:
T(n) = 3T(floor(n/4)) + Theta(n^2)
We need to show T(n) <= dn^2 for some d > 0.
Letting c be as before, we have:
T(n) <= 3T(floor(n/4)) + cn^2
<= 3d(floor(n/4))^2 + cn^2 by ind. hyp.
<= 3d(n/4)^2 + cn^2
= (3/16)dn^2 + cn^2
<= dn^2
if we choose d >= (16/13)c
For a second example we show 4.2.4
T(n) = O(n lg n) for the recurrence:
T(n) = T(n/3) + T(2n/3) + O(n), i.e.
T(n) = T(n/3) + T(2n/3) + cn.
Figure 4.2 (page 71) shows the recurrence tree
Cost
cn cn
______/ \_____
/ \
c(n/3) c(2n/3) cn
/ \ / \
/ \ / \
c(n/9) c(2n/9) c(2n/9) c(4n/9) cn
/ \ / \ / \ / \
/ \ / \ / \ / \
.
. ________________
Total: O(n lg n)
The longest path is down the right side, and
stops when n(2/3)^k = 1, i.e. k = log n
3/2
So the total cost seems to be bounded by
O(cn log_1.5(n)) = O(n lg n).
But there is a problem: we have not accounted
for the leaves. For a complete binary tree of
height log_1.5(n), there would be 2^log_1.5(n)
= n^log_1.5(2) leaves, each having a constant
cost, for a total of Theta(n^log_1.5(2)) which
is omega(n lg n). However, branches of the
tree "die out" faster on the left than on the
right, so there are < n^log_1.5(2) leaves.
4.2.5
It turns out that we don't have to do an
exact accounting of all nodes, since we can
prove T(n) = O(n lg n) by the substitution
method by showing that T(n) <= dn lg(n), for
some d > 0, as follows:
T(n) = T(n/3) + T(2n/3) + cn
<= d(n/3)lg(n/3) + d(2n/3)lg(2n/3) + cn
by ind. hyp.
= (d(n/3)lg(n) - d(n/3)lg(3))
+ (d(2n/3)lg(n) - d(2n/3)lg(3/2)) + cn
= dnlg(n) - d((n/3)lg(3) + (2n/3)lg(3/2))
+ cn
= dnlg(n)
- d((n/3)lg(3)+(2n/3)lg(3)-(2n/3)lg(2))
+ cn
= dnlg(n) - dn(lg(3) -2/3) +cn
<= dnlg(n)
if we choose d >= c/(lg(3) -2/3).
4.3 The master method 4.3.1
The master method solves recurrences of the
form: T(n) = aT(n/b)+f(n) for a >= 1 & b > 1.
Such a recurrence describes the running time
of an algorithm that divides a problem of
size n into a subproblems of size n/b. The a
subproblems are each solved in time T(n/b) and
f(n) is the cost of dividing the problem and
combining the solutions of the subproblems.
If we want to be precise, n/b may not be an
integer, but replacing it with floor(n/b) or
ceiling(n/b) doesn't affect the asymptotic
behavior of the solution, so we omit the floor
and ceiling functions for convenience.
The master theorem
Theorem 4.1 (Master theorem)
Let a >= 1 and b > 1, f(n) asymptotically
positive, and let T(n) be defined by:
T(n) = aT(n/b) + f(n)
where n/b can be interpreted to be either
floor(n/b) or ceiling(n/b). Then T(n) can be
bounded asymptotically as follows:
1. If f(n) = O(n^(log_b(a)-epsilon)) for some
epsilon > 0, then T(n) = Theta( n^log_b(a) )
2. If f(n) = Theta( n^log_b(a) ), then
T(n) = Theta( n^log_b(a) lg(n) )
3. If f(n) = Omega(n^(log_b(a)+epsilon)) for
some epsilon > 0, and if af(n/b) <= cf(n)
for some c < 1 and all n > n0 for some
n0 > 0, then T(n) = Theta( f(n) )
4.3.2
In this theorem, f(n) is compared with
n^log_b(a). In case 1, f(n) is polynomially
smaller (by a factor of n^epsilon), then the
recursive calls dominate, and the solution is
T(n) = Theta( n^log_b(a) ). In case 3, f(n)
is polynomially larger and satisfies the
"regularity" condition af(n/b) <= cf(n) for
some c < 1 and all n > n0 for some n0 > 0,
then f(n) dominates and T(n) = Theta(f(n)) is
the solution. Most of the polynomially
bounded functions we consider satisfy this
regularity condition. In case 2, f(n) and
n^log_b(a) are the same size, which adds a
lg(n) factor, so the solution is
T(n) = Theta( n^log_b(a) lg(n) ).
Note that these three cases don't cover all
the possibilities. There is a "gap" between
cases 1 and 2 when f(n) is smaller than
n^log_b(a), but not polynomially smaller.
There is a similar gap between cases 2 and 3
when f(n) is greater than n^log_b(a), but not
polynomially greater. And even when f(n) is
polynomially greater than n^log_b(a), the
master theorem cannot be applied if f(n) does
not satisfy the regularity condition.
Using the master method 4.3.3
As a first example, let T(n) = 9T(n/3) + n,
so a = 9, b = 3, f(n) = n, and n^log_b(a) =
n^log_3(9) = n^2. The master theorem's case 1
applies since f(n) = O(n^log_3(9)-epsilon),
where epsilon = 1 for instance, so that
T(n) = Theta(n^log_3(9)) = Theta(n^2).
As a second example, let T(n) = T(2n/3) + 1,
so a = 1, b = 3/2, f(n) = 1, and n^log_b(a) =
n^log_3/2(1) = n^0 = 1. The master theorem's
case 2 applies, since f(n) = Theta(n^log_b(a))
= Theta(1), so T(n) = Theta( 1 lg(n) ).
As a third example, let T(n) = 3T(n/4)+nlg(n)
so a = 3, b = 4, f(n) = nlg(n), & n^log_b(a) =
n^log_4(3) = O(n^0.793). Case 3 applies since
f(n) = Omega(n^log_4(3)+epsilon), epsilon
being about 0.2, and af(n/b) = 3(n/4)lg(n/4)
<= (3/4)nlg(n) = cf(n) for c = 3/4. Thus the
solution is T(n) = Theta( n lg(n) ).
Case 3 _seems_ to apply to the recurrence
T(n) = 2T(n/2) + nlg(n), which has the form
a = 2, b = 2, f(n) = nlg(n), and n^log_b(a) =
n^log_2(2) = n^1 = n, since f(n) = nlg(n) is
asymptotically larger than n^log_b(a) = n.
But it is not _polynomially_ larger, since
f(n)/n^log_b(a) = nlg(n)/n = lg(n) is
asymptotically less than n^epsilon for any
positive constant epsilon, and so falls into
the gap between cases 2 and 3; Exercise 4.4-2
shows the solution is T(n) = Theta(n lg^2(n)).