Chapter 7 Quicksort 7.1.1
Quicksort can sort an array of n numbers in
Theta(n lg n) time on average, but has a worst
case running time of Theta(n^2). However, it
sorts in place, and the constants in the Theta
notation are small, so it is used in practice.
Section 7.1 describes quicksort and the
partitioning algorithm upon which it depends.
Section 7.2 gives an intuitive analysis of
its running time, which is handled rigorously
in Section 7.4. Section 7.3 presents a
randomized version of quicksort which can
avoid the worst-case time of Theta(n^2) in
most situations.
7.1 Description of quicksort
Here is quicksort's 3-step divide-and-conquer
process for sorting a typical subarray A[p..r]
Divide: Partition A[p..r] into two (possibly
empty) subarrays A[p..q-1] and A[q+1..r]
such that each element of A[p..q-1] is <=
A[q], & A[q] is <= each element of A[q+1..r]
Compute q as part of the partition process.
Conquer: Sort A[p..q-1] and A[q+1..r] by
recursive calls to quicksort.
Combine: Since the subarrays are sorted in
place, the entire array A[p..r] is sorted.
QUICKSORT(A,p,r) 7.1.2
1 if p < r
2 then q <- PARTITION(A,p,r)
3 QUICKSORT(A,p,q-1)
4 QUICKSORT(A,q+1,r)
Partitioning the array
The key to QUICKSORT is PARTITION, which
rearranges the subarray A[p..r] in place.
Figure 7.1 page 147 shows how PARTITION works.
PARTITION(A,p,r)
1 x <- A[r]
2 i <- p - 1
3 for j <- p to r - 1
4 do if A[j] <= x
5 then i <- i + 1
6 exchange A[i] <-> A[j]
7 exchange A[i + 1] <-> A[r]
8 return i + 1
PARTITION selects an element x = A[r] as a
pivot element around which to partition the
subarray A[p..r], which is partitioned into 4
(possibly empty) regions, one of which is just
A[r] itself. At the start of each iteration
of the loop in lines 3-6 each region satisfies
certain properties, stated as a loop invariant
(summarized by Figure 7.2 on page 148).
For any array index k, 7.1.3
1. If p <= k <= i, then A[k] <= x
2. If i+1 <= k <= j-1, then A[k] > x
3. If k = r, then A[k] = x
The A[k] have no particular relationship to x
for j <= k < r.
Intialization: Prior to the first iteration,
i = p - 1, and j = p, so there are no index
values between p and i, or between i+1 & j-1
Maintenance: Figure 7.3 page 149 shows the two
possible outcomes of the "if" of line 4. If
A[j] > x (Figure 7.3(a)), condition 2 still
holds after j is incremented, and conditions
1 and 3 remain unchanged. If A[j] <= x
(Figure 7.3(b)), i is incremented, A[i] and
A[j] are swapped, and after j is incremented
1 and 2 still hold, and 3 is unchanged.
At Termination: j = r, so every entry is in
one of the three sets described by the
invariant: those <= x, those > x, & A[r] = x
Line 7 moves the pivot into place between
elements less than or equal to it and elements
greater than it, which is specified by the
"Divide" step.
The run time of PARTITION(A,p,r) is Theta(n),
where n = r - p + 1 (see Exercise 7.1-3).
7.2 Performance of quicksort 7.2.1
The performance of quicksort depends on how
well-balanced the partitioning is.
Worst-case partitioning
The worst case occurs when one of the sets
has n - 1 elements & the other has 0 elements.
If this happens in every call, we get the
recurrence T(n) = T(n - 1) + T(0) + Theta(n) =
T(n - 1) + Theta(n), since T(0) = Theta(1).
The recurrence has solution T(n) = Theta(n^2).
Note: this occurs when A is already sorted.
Best-case partitioning
The best case occurs if the sets are as close
to n/2 in size as possible (floor(n/2) and
ceiling(n/2) - 1). Then the recurrence is
T(n) = 2T(n/2) + Theta(n), which has solution
T(n) = Theta(n lg n) by the master theorem.
Balanced partitioning
Section 7.4 shows that the average-case run
time of quicksort is almost as good as the
best case (and far from the worst case),
namely Theta(n lg n).
As an example, suppose every partition 7.2.2
gave a 9-to-1 split, then we would obtain the
recurrence: T(n) <= T(9n/10) + T(n/10) + cn
for the running time. Figure 7.4, on page 151
shows the recurrence tree. All the levels
have cost cn down to log_10(n) where the left
branches start dying out, so the levels have
cost < cn below that. The tree height is
log (n), so the cost is < log (n) * cn
10/9 10/9
for non-leaves.
This example is very similar to the example
on pages 70-72, where it was also tricky to
evaluate the leaf costs. But, as in that
example, leaf costs can be ignored since
it can be proved directly by the substitution
method that T(n) = Theta(n lg n).
If every partition always produces the same
ratio of sizes of the sets, the run time is
still Theta(n lg n) - with a larger constant.
Intuition for the average case
If partitions alternate between "good" and
"bad", the non-recursive cost at each level is
Theta(n), as explained in Figure 7.5 page 152,
so the total cost is 2 x Theta(n lg n), since
the tree is just twice as high as one with
perfectly balanced partitioning.
7.3 A randomized version of quicksort 7.3.1
If the input to quicksort is random, we get
good performance, but we can't assume random
input. However, if we randomize the input
before doing the sorting, we get good average
case performance regardless of the original
input. Randomized versions of quicksort are
often considered to be the algorithm of choice
to sort large inputs.
Instead of randomizing all the input before
running quicksort, we can make random choices
of the pivot, which we choose randomly from
the subarray A[p..r]. We then exchange this
element with A[r] and run PARTITION as before.
Here are the randomized algorithms.
RANDOMIZED-PARTITION(A,p,r)
1 i <- RANDOM(p,r)
2 exchange A[r] <-> A[i]
3 return PARTITION(A,p,r)
RANDOMIZED-QUICKSORT(A,p,r)
1 if p < r
2 then q <- RANDOMIZED-PARTITION(A,p,r)
3 RANDOMIZED-QUICKSORT(A,p,q-1)
4 RANDOMIZED-QUICKSORT(A,q+1,r)
7.4 Analysis of quicksort 7.4.1
Section 7.2 gave intuitive reasons for worst,
best, and average case run times for quicksort
- here we do a more rigorous analysis of the
worst and average case run times (the best
case run time is analyzed in Exercise 7.4-2).
7.4.1 Worst-case analysis
From Section 7.2 we guess that the worst-case
run time of quicksort (& randomized quicksort)
is O(n^2), and we prove that here using the
substitution method.
Letting T(n) be the worst-case time for
quicksort, we have the recurrence:
T(n) = max (T(q) + T(n-1 - q)) + Theta(n)
0<=q<=n-1
Substituting our guess that T(n) <= cn^2 gives
T(n) <= max (cq^2 + c(n-1 - q)^2) + Theta(n)
0<=q<=n-1
= c max (q^2 + (n-1 - q)^2) + Theta(n)
0<=q<=n-1
f(q) = q^2 + (n-1 - q)^2 achieves its maximum
value (n-1)^2 at either endpoint of [0,n-1]
since the second derivative f"(q) > 0, so
T(n) <= c(n-1)^2 + Theta(n)
<= cn^2 - c(2n - 1) + kn 7.4.2
<= cn^2 - cn + kn (for n >= 1)
<= cn^2 (if we choose c > k)
Thus T(n) = O(n^2). Similarly, it can be
proved that T(n) = Omega(n^2) (Exercise 7.4-1)
so that T(n) = Theta(n^2)
7.4.2 Expected running time
We prove that the average-case run time, T(n),
for randomized quicksort (or quicksort with
average/random input) is O(n lg n), which when
combined with the intuitive best-case bound
of Omega(n lg n) from Section 7.2 gives
Theta(n lg n) expected running time (it can
also be proved directly that T(n) =
Omega(n lg n) as suggested in Exercise 7.4-4).
Running time and comparisons
The run time of quicksort is clearly dominated
by the time spent in the PARTITION procedure.
Note that once a pivot element is chosen, it
is never accessed again by any recursive calls
to QUICKSORT or PARTITION. Thus there can be
at most n calls to PARTITION - in fact at most
n-1 calls on sorted input.
The fewest calls to PARTITION occur when 7.4.3
every other element in the sorted array was a
pivot - otherwise there would be a subarray of
size >= 2 which would need to be sorted and
another pivot picked (giving a contradiction).
So the number of calls to PARTITION is also
>= floor(n/2), and thus is Theta(n).
Each call to PARTITION takes O(1) time plus
the number of iterations of its for-loop,
which can be counted by the number of
comparisons, X, in line 4. Thus we have:
Lemma 7.1
The running time of QUICKSORT is Theta(n + X)
Thus if we show that the value of X on
random/average input is O(n lg n), this will
dominate the n term and show that the average-
case run time is also O(n lg n) (and therefore
is also Theta(n lg n) - using Exercise 7.4-4).
For notational ease in the analysis, we rename
the elements of A according to their sorted
(increasing) order as z_1, z_2, ... , z_n.
Also, let Z_ij = {z_i, z_(i+1), ... , z_j}.
We note that any pair of elements is compared
at most once - when one of them is a pivot
(remember that after an element is a pivot, it
is never compared with any element again).
Let X_ij = I{z_i is compared to z_j}, the
indicator random variable equal to 1 if z_i is
compared to z_j, and equal to 0 if not.
Since each pair is compared at most 7.4.4
once, the total number of comparisons is:
n-1 n
X = Sum Sum X_ij
i=1 j=i+1
Taking expectations of both sides and using
linearity of expectation (C.20 page 1108):
n-1 n
E[X] = E[ Sum Sum X_ij ]
i=1 j=i+1
n-1 n
= Sum Sum E[X_ij] (by linearity of E[])
i=1 j=i+1
n-1 n
= Sum Sum Pr{z_i is compared to z_j} (*)
i=1 j=i+1
where the last equations is true by Lemma 5.1
(page 95), which says that the expected value
of an indicator random variable for an event A
is just the probability Pr{A} that A occurs.
To calculate the probability in (*), we note
that once a partition is made, no element from
the upper partition is ever compared with any
element from the lower partition. So once a
pivot x is chosen with z_i < x < z_j, z_i and
z_j will never be compared.
However if z_i is chosen as a pivot 7.4.5
before any other element in Z_ij, then z_i
will be compared with z_j (and every other
element in Z_ij except itself). Similarly, if
z_j is chosen as a pivot before any other
element in Z_ij, then z_j will be compared
with z_i (and every other element in Z_ij
except itself). Thus z_i is compared to z_j
if and only if z_i or z_j is the first element
to be chosen as a pivot from Z_ij.
To calculate the probability, we note that
prior to the time the first pivot was chosen
from Z_ij, all of Z_ij must have been in the
same partition. Therefore all the elements of
Z_ij are equally likely to be chosen as the
first pivot from Z_ij. Since the set Z_ij has
j - i + 1 elements, the probability that any
given element is chosen as the first pivot is
1/(j - i + 1). Thus:
Pr{z_i is compared to z_j}
= Pr{z_i or z_j is the first pivot from Z_ij}
= Pr{z_i is first pivot chosen from Z_ij}
+ Pr{z_j is first pivot chosen from Z_ij}
= 1/(j - i + 1) + 1/(j - i + 1)
= 2/(j - i + 1) (**)
The second equality follows since the events
are independent.
Combining (*) and (**), we get: 7.4.6
n-1 n
E[X] = Sum Sum 2/(j - i + 1)
i=1 j=i+1
Making the substitution k = j-i, we get:
n-1 n-i
E[X] = Sum Sum 2/(k + 1)
i=1 k=1
n n
< Sum Sum 2/k
i=1 k=1
n
= Sum O(lg n) (by A.7 page 1060, which
i=1 follows from A.14 p. 1067)
= O(n lg n)
Which proves that random quicksort or
quicksort on random/average input runs in
O(n lg n) time on the average (actually
Theta(n lg n) when combined with Exercise
7.4-4 on page 159).