Chapter 7 Quicksort 7.1.1 Quicksort can sort an array of n numbers in Theta(n lg n) time on average, but has a worst case running time of Theta(n^2). However, it sorts in place, and the constants in the Theta notation are small, so it is used in practice. Section 7.1 describes quicksort and the partitioning algorithm upon which it depends. Section 7.2 gives an intuitive analysis of its running time, which is handled rigorously in Section 7.4. Section 7.3 presents a randomized version of quicksort which can avoid the worst-case time of Theta(n^2) in most situations. 7.1 Description of quicksort Here is quicksort's 3-step divide-and-conquer process for sorting a typical subarray A[p..r] Divide: Partition A[p..r] into two (possibly empty) subarrays A[p..q-1] and A[q+1..r] such that each element of A[p..q-1] is <= A[q], & A[q] is <= each element of A[q+1..r] Compute q as part of the partition process. Conquer: Sort A[p..q-1] and A[q+1..r] by recursive calls to quicksort. Combine: Since the subarrays are sorted in place, the entire array A[p..r] is sorted. QUICKSORT(A,p,r) 7.1.2 1 if p < r 2 q = PARTITION(A,p,r) 3 QUICKSORT(A,p,q-1) 4 QUICKSORT(A,q+1,r) Partitioning the array The key to QUICKSORT is PARTITION, which rearranges the subarray A[p..r] in place. Figure 7.1 shows how PARTITION works. PARTITION(A,p,r) 1 x = A[r] 2 i = p - 1 3 for j = p to r - 1 4 if A[j] <= x 5 i = i + 1 6 exchange A[i] <-> A[j] 7 exchange A[i + 1] <-> A[r] 8 return i + 1 PARTITION selects an element x = A[r] as a pivot element around which to partition the subarray A[p..r], which is partitioned into 4 (possibly empty) regions, one of which is just A[r] itself. At the start of each iteration of the loop in lines 3-6 each region satisfies certain properties, stated as a loop invariant (summarized by Figure 7.2). For any array index k, 7.1.3 1. If p <= k <= i, then A[k] <= x 2. If i+1 <= k <= j-1, then A[k] > x 3. If k = r, then A[k] = x The A[k] have no particular relationship to x for j <= k < r. Intialization: Prior to the first iteration, i = p - 1, and j = p, so there are no index values between p and i, or between i+1 & j-1 Maintenance: Figure 7.3 shows the two possible outcomes of the "if" of line 4. If A[j] > x (Figure 7.3(a)), condition 2 still holds after j is incremented, and conditions 1 and 3 remain unchanged. If A[j] <= x (Figure 7.3(b)), i is incremented, A[i] and A[j] are swapped, and after j is incremented 1 and 2 still hold, and 3 is unchanged. At Termination: j = r, so every entry is in one of the three sets described by the invariant: those <= x, those > x, & A[r] = x Line 7 moves the pivot into place between elements less than or equal to it and elements greater than it, which is specified by the "Divide" step. The run time of PARTITION(A,p,r) is Theta(n), where n = r - p + 1 (see Exercise 7.1-3). 7.2 Performance of quicksort 7.2.1 The performance of quicksort depends on how well-balanced the partitioning is. Worst-case partitioning The worst case occurs when one of the sets has n - 1 elements & the other has 0 elements. If this happens in every call, we get the recurrence T(n) = T(n - 1) + T(0) + Theta(n) = T(n - 1) + Theta(n), since T(0) = Theta(1). The recurrence has solution T(n) = Theta(n^2). Note: this occurs when A is already sorted. Best-case partitioning The best case occurs if the sets are as close to n/2 in size as possible (floor(n/2) and ceiling(n/2) - 1). Then the recurrence is T(n) = 2T(n/2) + Theta(n), which has solution T(n) = Theta(n lg n) by the master theorem. Balanced partitioning Section 7.4 shows that the average-case run time of quicksort is almost as good as the best case (and far from the worst case), namely Theta(n lg n). As an example, suppose every partition 7.2.2 gave a 9-to-1 split, then we would obtain the recurrence: T(n) <= T(9n/10) + T(n/10) + cn for the running time. Figure 7.4 shows the recurrence tree. All the levels have cost cn down to log_10(n) where the left branches start dying out, so the levels have cost < cn below that. The tree height is log (n), so the cost is < log (n) * cn 10/9 10/9 for non-leaves. This example is very similar to the example Figure 4.6, where it was also tricky to evaluate the leaf costs. But, as in that example, leaf costs can be ignored since it can be proved directly by the substitution method that T(n) = Theta(n lg n). If every partition always produces the same ratio of sizes of the sets, the run time is still Theta(n lg n) - with a larger constant. Intuition for the average case If partitions alternate between "good" and "bad", the non-recursive cost at each level is Theta(n), as explained in Figure 7.5, the total cost is 2 x Theta(n lg n), since the tree is just twice as high as one with perfectly balanced partitioning. 7.3 A randomized version of quicksort 7.3.1 If the input to quicksort is random, we get good performance, but we can't assume random input. However, if we randomize the input before doing the sorting, we get good average case performance regardless of the original input. Randomized versions of quicksort are often considered to be the algorithm of choice to sort large inputs. Instead of randomizing all the input before running quicksort, we can make random choices of the pivot, which we choose randomly from the subarray A[p..r]. We then exchange this element with A[r] and run PARTITION as before. Here are the randomized algorithms. RANDOMIZED-PARTITION(A,p,r) 1 i = RANDOM(p,r) 2 exchange A[r] <-> A[i] 3 return PARTITION(A,p,r) RANDOMIZED-QUICKSORT(A,p,r) 1 if p < r 2 q = RANDOMIZED-PARTITION(A,p,r) 3 RANDOMIZED-QUICKSORT(A,p,q-1) 4 RANDOMIZED-QUICKSORT(A,q+1,r) 7.4 Analysis of quicksort 7.4.1 Section 7.2 gave intuitive reasons for worst, best, and average case run times for quicksort - here we do a more rigorous analysis of the worst and average case run times (the best case run time is analyzed in Exercise 7.4-2). 7.4.1 Worst-case analysis From Section 7.2 we guess that the worst-case run time of quicksort (& randomized quicksort) is O(n^2), and we prove that here using the substitution method. Letting T(n) be the worst-case time for quicksort, we have the recurrence: T(n) = max (T(q) + T(n-1 - q)) + Theta(n) 0<=q<=n-1 Substituting our guess that T(n) <= cn^2 gives T(n) <= max (cq^2 + c(n-1 - q)^2) + Theta(n) 0<=q<=n-1 = c max (q^2 + (n-1 - q)^2) + Theta(n) 0<=q<=n-1 f(q) = q^2 + (n-1 - q)^2 achieves its maximum value (n-1)^2 at either endpoint of [0,n-1] since the second derivative f"(q) > 0, so T(n) <= c(n-1)^2 + Theta(n) <= cn^2 - c(2n - 1) + kn 7.4.2 <= cn^2 - cn + kn (for n >= 1) <= cn^2 (if we choose c > k) Thus T(n) = O(n^2). Similarly, it can be proved that T(n) = Omega(n^2) (Exercise 7.4-1) so that T(n) = Theta(n^2) 7.4.2 Expected running time We prove that the average-case run time, T(n), for randomized quicksort (or quicksort with average/random input) is O(n lg n), which when combined with the intuitive best-case bound of Omega(n lg n) from Section 7.2 gives Theta(n lg n) expected running time (it can also be proved directly that T(n) = Omega(n lg n) as suggested in Exercise 7.4-4). Running time and comparisons The run time of quicksort is clearly dominated by the time spent in the PARTITION procedure. Note that once a pivot element is chosen, it is never accessed again by any recursive calls to QUICKSORT or PARTITION. Thus there can be at most n calls to PARTITION - in fact at most n-1 calls on sorted input. The fewest calls to PARTITION occur when 7.4.3 every other element in the sorted array was a pivot - otherwise there would be a subarray of size >= 2 which would need to be sorted and another pivot picked (giving a contradiction). So the number of calls to PARTITION is also >= floor(n/2), and thus is Theta(n). Each call to PARTITION takes O(1) time plus the number of iterations of its for-loop, which can be counted by the number of comparisons, X, in line 4. Thus we have: Lemma 7.1 The running time of QUICKSORT is Theta(n + X) Thus if we show that the value of X on random/average input is O(n lg n), this will dominate the n term and show that the average- case run time is also O(n lg n) (and therefore is also Theta(n lg n) - using Exercise 7.4-4). For notational ease in the analysis, we rename the elements of A according to their sorted (increasing) order as z_1, z_2, ... , z_n. Also, let Z_ij = {z_i, z_(i+1), ... , z_j}. We note that any pair of elements is compared at most once - when one of them is a pivot (remember that after an element is a pivot, it is never compared with any element again). Let X_ij = I{z_i is compared to z_j}, the indicator random variable equal to 1 if z_i is compared to z_j, and equal to 0 if not. Since each pair is compared at most 7.4.4 once, the total number of comparisons is: n-1 n X = Sum Sum X_ij i=1 j=i+1 Taking expectations of both sides and using linearity of expectation (C.21 page 1198): n-1 n E[X] = E[ Sum Sum X_ij ] i=1 j=i+1 n-1 n = Sum Sum E[X_ij] (by linearity of E[]) i=1 j=i+1 n-1 n = Sum Sum Pr{z_i is compared to z_j} (*) i=1 j=i+1 where the last equations is true by Lemma 5.1 (page 118), which says that the expected value of an indicator random variable for an event A is just the probability Pr{A} that A occurs. To calculate the probability in (*), we note that once a partition is made, no element from the upper partition is ever compared with any element from the lower partition. So once a pivot x is chosen with z_i < x < z_j, z_i and z_j will never be compared. However if z_i is chosen as a pivot 7.4.5 before any other element in Z_ij, then z_i will be compared with z_j (and every other element in Z_ij except itself). Similarly, if z_j is chosen as a pivot before any other element in Z_ij, then z_j will be compared with z_i (and every other element in Z_ij except itself). Thus z_i is compared to z_j if and only if z_i or z_j is the first element to be chosen as a pivot from Z_ij. To calculate the probability, we note that prior to the time the first pivot was chosen from Z_ij, all of Z_ij must have been in the same partition. Therefore all the elements of Z_ij are equally likely to be chosen as the first pivot from Z_ij. Since the set Z_ij has j - i + 1 elements, the probability that any given element is chosen as the first pivot is 1/(j - i + 1). Thus: Pr{z_i is compared to z_j} = Pr{z_i or z_j is the first pivot from Z_ij} = Pr{z_i is first pivot chosen from Z_ij} + Pr{z_j is first pivot chosen from Z_ij} = 1/(j - i + 1) + 1/(j - i + 1) = 2/(j - i + 1) (**) The second equality follows since the events are independent. Combining (*) and (**), we get: 7.4.6 n-1 n E[X] = Sum Sum 2/(j - i + 1) i=1 j=i+1 Making the substitution k = j-i, we get: n-1 n-i E[X] = Sum Sum 2/(k + 1) i=1 k=1 n n < Sum Sum 2/k i=1 k=1 n = Sum O(lg n) (by A.7 page 1060, which i=1 follows from A.14 p. 1067) = O(n lg n) Which proves that random quicksort or quicksort on random/average input runs in O(n lg n) time on the average (actually Theta(n lg n) when combined with Exercise 7.4-4 on page 159).