Chapter 4 Recurrences 4.0.1 The running time of a divide-and-conquer algorithm can often be described by a Recurrence: an equation or inequality that describes a function in terms of its value on smaller inputs. For example, the running time of merge-sort: T(n) = / Theta(1) if n = 1 \ 2T(n/2) + Theta(n) if n > 1 Three methods for solving recurrences -- i.e. obtaining Theta or big-O bounds on T(n): 1) The substitution method: make a guess and then prove the guess is correct by induction. 2) The recursion-tree method: convert the recurrence into a tree of costs & sum them. 3) The "master method": gives bounds for recurrences of the form: T(n) = aT(n/b)+f(n) where a >= 1, b > 1, and f(n) is given. Technicalities 4.0.2 1. Usually the running time T(n) is only defined for integer n. This is often ignored in practice. For instance, the precise recurrence for the run time of merge sort is: T(n) = / Theta(1) if n = 1 \ T(floor(n/2)) + T(ceiling(n/2)) + Theta(n) if n > 1 2. We usually don't give boundary conditions, since T(n) = Theta(1) for small n. Usually changing T(1) only changes the solution to the recurrence by a constant factor. So, for example, we usually just state the recurrence for the run time of merge sort simply as: T(n) = 2T(n/2) + Theta(n) Thus we usually omit floors, ceilings, and boundary conditions when we solve recurrences. Usually they don't matter, but we check to make sure that that is the case after we have obtained our solution. 4.1 The substitution method 4.1.1 The substitution method involves two steps: 1. Guess the form of the solution. 2. Use mathematical induction to find the constants and show that the solution works. This method can establish upper or lower bounds or both on a recurrence. For example, we determine an upper bound on the recurrence: T(n) = 2T(floor(n/2)) + n (4.4) 1. We guess the solution is T(n) = O(n lg n), so we need to prove 2. T(n) <= cn lg(n) for an appropriate choice of c. We start by assuming this holds for floor(n/2), i.e. that T(floor(n/2)) <= cfloor(n/2) lg(floor(n/2)) Substituting this into (4.4) above gives: T(n) <= 2[cfloor(n/2) lg(floor(n/2))] + n <= cn lg(n/2) + n = cn lg(n) - cn lg(2) + n = cn lg(n) - cn + n <= cn lg(n) where the last step holds if c >= 1. To complete the inductive proof, we need to pick c so that the inequality holds for the boundary case too. Now T(1) > 0, but lg(1) is 0, which seems to be a problem. 4.1.2 But we only need T(n) <= cn lg(n) for n >= n0, for some n0 > 0. We can let n0 = 2 and pick c so that T(2) <= c 2 lg(2) and T(3) < c 3 lg(3) to establish the base cases n = 2 and n = 3. Since T(2) = 2T(floor(2/2)) + 2 = 2T(1) + 2, and T(3) = 2T(floor(3/2)) + 3 = 2T(1) + 3, then letting c = maximum of (2T(1)+2)/(2lg2) and (2T(1)+3)/(3lg3) will work. The text shows that if T(1) = 1, then c = 2. Making a good guess Of course guessing is an art. But here are two methods: 1. If a recurrence is similar to one you have solved before, the solution to the new recurrence will be similar to the old one. Consider for example the recurrence: T(n) = 2T(floor(n/2) + 17) + n which is similar to (4.4) above. So you might correctly guess that T(n) = O(n lg n), which can be proved by the substitution method. 2. Guess easy upper and lower bounds, such as T(n) = Omega(n) and T(n) = O(n^2) for (4.4), and then improve them until they converge to the asymptotically tight T(n) = Theta(n lg n). Subtleties 4.1.3 A trick may be needed to make the solution go through -- subtracting a lower order term for example. Consider the recurrence: T(n) = T(floor(n/2)) + T(ceiling(n/2)) + 1 It is natural to guess the solution is O(n). But trying to prove T(n) <= cn leads to: T(n) <= c floor(n/2) + c ceiling(n/2) + 1 = c n + 1 which does not give T(n) <= c n for any choice of c. Now T(n) = O(n) is correct, but to show it, we have to guess that T(n) <= c n - b, so: T(n) <= c floor(n/2) - b + c ceiling(n/2) - b + 1 = c n - 2 b + 1 <= c n - b if we let b >= 1. As before, we must also choose c to handle the boundary conditions. Avoiding pitfalls It is easy to make mistakes. For example, we can "prove" T(n) = O(n) for (4.4) by guessing T(n) <= cn and then arguing: T(n) <= 2 c floor(n/2) + n <= c n + n = O(n) <=== Wrong!! The error is that we haven't proved the exact form of the inductive hypothesis: T(n) <= cn. Changing variables 4.1.4 Sometimes algebraic manipulation can make an unknown recurrence look like one you have solved. For example, consider the recurrence: T(n) = 2T(floor(sqrt(n))) + lg n Ignoring round-to-integer problems, we let m = lg(n), so that n = 2^m, and we get: T(2^m) = 2T(2^(m/2)) + m And now, letting S(m) = T(2^m), we have: S(m) = 2S(m/2) + m which we suspect (and can show) has the solution S(m) = O(m lg m). Changing back to T(n), T(n) = T(2^m) = S(m) = O(m lg m) = O(lg n lg lg n). 4.2 The recursion-tree method 4.2.1 In a recursion tree, each node represents the cost of a recursive call of the algorithm somewhere in its recursive invocation, but ignoring the cost of further recursive calls from within it, which are accounted for by "child" costs of that node. Once the costs of each node have been determined, the costs of the nodes at each level are summed, and then the per-level costs are summed to get the final answer. Recursion trees are especially useful in finding the running time of a divide-and-conquer algorithm Recursion trees are often used to generate good guesses, which can then be verified by the substitution method. Used in this way, we can tolerate a bit of "sloppiness" in the calculations. Or, by being careful, we can turn the guess into a proof (which is done in Section 4.4 when proving the master method). We first use a recursion tree to guess a solution to the recurrence: T(n) = 3T(floor(n/4)) + Theta(n^2) We want to find an upper bound for T(n). We ignore the floor function and non-integer values of n/4, so we can change the recurrence to: T(n) = 3T(n/4) + cn^2 for some c > 0. Figure 4.1, page 69, shows the construction of the recursion tree for this recurrence. 4.2.2 Figure 4.1(b) shows one expansion of T(n) into the non-recursive cost, cn^2, of the root and the three recursive costs, T(n/4), of its children: cn^2 / | \ / | \ / | \ / | \ / | \ T(n/4) T(n/4) T(n/4) Figure 4.1(c) shows the expansion of T(n) one level deeper, with each of the T(n/4) costs replaced by the non-recursive cost, c(n/4)^2, and the three recursive costs, T(n/16), of its children. We continue this process until we reach the boundary condition, a subproblem of size 1. If this occurs at depth i, n/4^i = 1, i.e. i = log_4(n). Thus the tree has log_4(n) + 1 levels: 0, 1, 2, ..., log_4(n), as is shown in Figure 4.1(d). To get the per-level cost, we note that the non-recursive cost at each level decreases by a factor of 4 and there are 3 times as many of them as in the previous level. So node cost at level i is c(n/4^i)^2 and there are 3^i such costs for a total of (3/16)^i cn^2. At the last level of depth log_4(n), there are 3^log_4(n) = n^log_4(3) nodes of cost T(1) for a total cost of Theta(n^log_4(3)). Thus the total cost of all levels is: 4.2.3 log_4(n)-1 T(n) = Sum ( (3/16)^i cn^2 ) + i=0 Theta(n^log_4(3)) inf < Sum( (3/16)^i cn^2) + Theta(n^log_4(3)) i=0 = 1/(1 - 3/16)cn^2 + Theta(n^log_4(3)) = 16/13 cn^2 + Theta(n^log_4(3)) = O(n^2) Which gives us our guess. Note that 13/13 of the 16/13 of the cn^2 term comes from the root and so T(n) = Omega(n^2), since it must take at least as much time to run as for the root. We now use the substitution method to verify our guess, T(n) = O(n^2) as an upper bound for the original recurrence: T(n) = 3T(floor(n/4)) + Theta(n^2) We need to show T(n) <= dn^2 for some d > 0. Letting c be as before, we have: T(n) <= 3T(floor(n/4)) + cn^2 <= 3d(floor(n/4))^2 + cn^2 by ind. hyp. <= 3d(n/4)^2 + cn^2 = (3/16)dn^2 + cn^2 <= dn^2 if we choose d >= (16/13)c For a second example we show 4.2.4 T(n) = O(n lg n) for the recurrence: T(n) = T(n/3) + T(2n/3) + O(n), i.e. T(n) = T(n/3) + T(2n/3) + cn. Figure 4.2 (page 71) shows the recurrence tree Cost cn cn ______/ \_____ / \ c(n/3) c(2n/3) cn / \ / \ / \ / \ c(n/9) c(2n/9) c(2n/9) c(4n/9) cn / \ / \ / \ / \ / \ / \ / \ / \ . . ________________ Total: O(n lg n) The longest path is down the right side, and stops when n(2/3)^k = 1, i.e. k = log n 3/2 So the total cost seems to be bounded by O(cn log_1.5(n)) = O(n lg n). But there is a problem: we have not accounted for the leaves. For a complete binary tree of height log_1.5(n), there would be 2^log_1.5(n) = n^log_1.5(2) leaves, each having a constant cost, for a total of Theta(n^log_1.5(2)) which is omega(n lg n). However, branches of the tree "die out" faster on the left than on the right, so there are < n^log_1.5(2) leaves. 4.2.5 It turns out that we don't have to do an exact accounting of all nodes, since we can prove T(n) = O(n lg n) by the substitution method by showing that T(n) <= dn lg(n), for some d > 0, as follows: T(n) = T(n/3) + T(2n/3) + cn <= d(n/3)lg(n/3) + d(2n/3)lg(2n/3) + cn by ind. hyp. = (d(n/3)lg(n) - d(n/3)lg(3)) + (d(2n/3)lg(n) - d(2n/3)lg(3/2)) + cn = dnlg(n) - d((n/3)lg(3) + (2n/3)lg(3/2)) + cn = dnlg(n) - d((n/3)lg(3)+(2n/3)lg(3)-(2n/3)lg(2)) + cn = dnlg(n) - dn(lg(3) -2/3) +cn <= dnlg(n) if we choose d >= c/(lg(3) -2/3). 4.3 The master method 4.3.1 The master method solves recurrences of the form: T(n) = aT(n/b)+f(n) for a >= 1 & b > 1. Such a recurrence describes the running time of an algorithm that divides a problem of size n into a subproblems of size n/b. The a subproblems are each solved in time T(n/b) and f(n) is the cost of dividing the problem and combining the solutions of the subproblems. If we want to be precise, n/b may not be an integer, but replacing it with floor(n/b) or ceiling(n/b) doesn't affect the asymptotic behavior of the solution, so we omit the floor and ceiling functions for convenience. The master theorem Theorem 4.1 (Master theorem) Let a >= 1 and b > 1, f(n) asymptotically positive, and let T(n) be defined by: T(n) = aT(n/b) + f(n) where n/b can be interpreted to be either floor(n/b) or ceiling(n/b). Then T(n) can be bounded asymptotically as follows: 1. If f(n) = O(n^(log_b(a)-epsilon)) for some epsilon > 0, then T(n) = Theta( n^log_b(a) ) 2. If f(n) = Theta( n^log_b(a) ), then T(n) = Theta( n^log_b(a) lg(n) ) 3. If f(n) = Omega(n^(log_b(a)+epsilon)) for some epsilon > 0, and if af(n/b) <= cf(n) for some c < 1 and all n > n0 for some n0 > 0, then T(n) = Theta( f(n) ) 4.3.2 In this theorem, f(n) is compared with n^log_b(a). In case 1, f(n) is polynomially smaller (by a factor of n^epsilon), then the recursive calls dominate, and the solution is T(n) = Theta( n^log_b(a) ). In case 3, f(n) is polynomially larger and satisfies the "regularity" condition af(n/b) <= cf(n) for some c < 1 and all n > n0 for some n0 > 0, then f(n) dominates and T(n) = Theta(f(n)) is the solution. Most of the polynomially bounded functions we consider satisfy this regularity condition. In case 2, f(n) and n^log_b(a) are the same size, which adds a lg(n) factor, so the solution is T(n) = Theta( n^log_b(a) lg(n) ). Note that these three cases don't cover all the possibilities. There is a "gap" between cases 1 and 2 when f(n) is smaller than n^log_b(a), but not polynomially smaller. There is a similar gap between cases 2 and 3 when f(n) is greater than n^log_b(a), but not polynomially greater. And even when f(n) is polynomially greater than n^log_b(a), the master theorem cannot be applied if f(n) does not satisfy the regularity condition. Using the master method 4.3.3 As a first example, let T(n) = 9T(n/3) + n, so a = 9, b = 3, f(n) = n, and n^log_b(a) = n^log_3(9) = n^2. The master theorem's case 1 applies since f(n) = O(n^log_3(9)-epsilon), where epsilon = 1 for instance, so that T(n) = Theta(n^log_3(9)) = Theta(n^2). As a second example, let T(n) = T(2n/3) + 1, so a = 1, b = 3/2, f(n) = 1, and n^log_b(a) = n^log_3/2(1) = n^0 = 1. The master theorem's case 2 applies, since f(n) = Theta(n^log_b(a)) = Theta(1), so T(n) = Theta( 1 lg(n) ). As a third example, let T(n) = 3T(n/4)+nlg(n) so a = 3, b = 4, f(n) = nlg(n), & n^log_b(a) = n^log_4(3) = O(n^0.793). Case 3 applies since f(n) = Omega(n^log_4(3)+epsilon), epsilon being about 0.2, and af(n/b) = 3(n/4)lg(n/4) <= (3/4)nlg(n) = cf(n) for c = 3/4. Thus the solution is T(n) = Theta( n lg(n) ). Case 3 _seems_ to apply to the recurrence T(n) = 2T(n/2) + nlg(n), which has the form a = 2, b = 2, f(n) = nlg(n), and n^log_b(a) = n^log_2(2) = n^1 = n, since f(n) = nlg(n) is asymptotically larger than n^log_b(a) = n. But it is not _polynomially_ larger, since f(n)/n^log_b(a) = nlg(n)/n = lg(n) is asymptotically less than n^epsilon for any positive constant epsilon, and so falls into the gap between cases 2 and 3; Exercise 4.4-2 shows the solution is T(n) = Theta(n lg^2(n)).