Chapter 17 Amortized Analysis 17.0.1
In amortized analysis, the time required for
a sequence of operations is averaged. This
average may be low even if a single operation
in the sequence is expensive. Amortized
analysis differs from average-case analysis in
that probability is not involved; amortized
analysis guarantees the average performance of
each operation in the worst case.
Sections 17.1, 17.2, & 17.3 cover the 3 most
common methods used in amortized analysis.
Section 17.1 treats aggregate analysis, which
determines an upper bound on the total cost
T(n) of n operations. Each operation is given
the average T(n)/n as its amortized cost.
Section 17.2 treats the accounting method,
in which we determine an amortized cost for
each operation. This method overcharges some
operations early in the sequence, storing the
overcharge as a "prepaid credit" to be used to
pay for later operations that are charged less
than they actually cost.
Section 17.3 treats the potential method, in
which we also determine an amortized cost for
each operation, and may overcharge operations
early to make up for later undercharges. The
credit is maintained as the "potential energy"
of the data structure as a whole instead of
being assigned to individual objects in it.
17.1 Aggregate analysis 17.1.1
In aggregate analysis, we show that for all n
a sequence of n operations takes worst-case
time T(n) in total, so that the average or
amortized cost for each operation is T(n)/n.
Stack operations
As a first example of aggregate analysis, we
analyze stacks that also have a MULTIPOP
operation, in addition to PUSH(S,x) and POP(S)
which take O(1) time, and which we assume to
have cost 1, so that the cost of n PUSH and
POP operations is n, and hence Theta(n).
MULTIPOP(S,k) pops the k top objects from S
or pops the whole stack if it has fewer than k
objects. It uses the STACK-EMPTY function.
Figure 17.1(a) shows a stack; Figure 17.1(b)
shows it after MULTIPOP(S,4); & Figure 17.1(c)
shows the empty stack after MULTIPOP(S,7):
top -> 23 Figure 17.1
17
6
39
10 top --> 10
47 47
--- --- --- (empty)
(a) (b) (c)
MULTIPOP(S,k) 17.1.2
1 while not STACK-EMPTY(S) and k not = 0
2 POP(S)
3 k = k - 1
The running time of MULTIPOP(S,k) on a stack
of s objects is O(k) since the while loop
iterates min(s,k) times and each operation
takes Theta(1) time.
Let us analyze a sequence of n PUSH, POP, and
MULTIPOP operations on an initially empty
stack. The worst-case cost of a MULTIPOP in
the sequence is O(n) since the stack size is
at most n. Thus the worst-case time of any
stack operation is O(n), and hence a sequence
of n operations could cost O(n^2). But this
bound is not tight.
We use aggregate analysis to obtain a better
bound. Note that any sequence of n operations
on an empty stack can cost at most O(n). Why?
Because each object can be popped at most once
for each time it is pushed, and so the number
of pops, including calls within MULTIPOP is at
most the number of pushes, which is at most n.
So the average cost of an operation is O(n)/n
= O(1), the amortized cost of each operation.
Note: no probability was involved. We found
O(n) to be the worst-case bound on the cost of
a sequence of n operations; dividing by n is
the average or amortized cost per operation.
Incrementing a binary counter 17.1.3
As a second example of aggregate analysis, we
consider a k-bit binary counter that counts up
from 0. The counter is an array A[0..k-1] of
bits, where A.length = k. A binary number x
has lowest-order bit in A[0] and highest-order
bit in A[k-1], so that: k-1 i
x = Sum ( A[i] * 2 )
i = 0
Initially, x = 0 so A[i] = 0 for all i. To
add 1 (modulo 2^k), to the counter, use:
INCREMENT(A)
1 i = 0
2 while i < A.length and A[i] = 1
3 A[i] = 0
4 i = i + 1
5 if i < A.length
6 A[i] = 1
At the start of each iteration of the while
loop, we wish to add 1 into position i. If
A[i] = 1, we flip the bit to 0 in A[i] which
yields a carry of 1 to be added into position
i+1. Otherwise the loop ends, and if i < k,
A[i] = 0, so adding 1 to that position flips
A[i] to 1. The cost of each call to INCREMENT
is linear in the number of bits flipped.
Figure 17.2 (page 455) below 17.1.4
shows what happens when a binary counter is
incremented 16 times starting at 0.
Counter Total
value A[6] A[5] A[4] A[3] A[2] A[1] A[0] cost
0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 (1) 1
2 0 0 0 0 0 (1 0) 3
3 0 0 0 0 0 1 (1) 4
4 0 0 0 0 (1 0 0) 7
5 0 0 0 0 1 0 (1) 8
6 0 0 0 0 1 (1 0) 10
7 0 0 0 0 1 1 (1) 11
8 0 0 0 (1 0 0 0) 15
9 0 0 0 1 0 0 (1) 16
10 0 0 0 1 0 (1 0) 18
11 0 0 0 1 0 1 (1) 19
12 0 0 0 1 (1 0 0) 22
13 0 0 0 1 1 0 (1) 23
14 0 0 0 1 1 (1 0) 25
15 0 0 0 1 1 1 (1) 26
16 0 0 (1 0 0 0 0) 31
Note that this is slightly different than
Figure 17.2 in that the bits that are flipped
to achieve _this_ counter value are enclosed
in parentheses. Note also that the total cost
is always less than twice the number of
INCREMENT operations.
17.1.5
Note that INCREMENT takes Theta(k) time in
the worst case when A contains all 1's. Thus
a sequence of n INCREMENT operations on a
counter initially at 0 could take O(nk) time.
As in the stack example, we can tighten this
bound to O(n) by observing that not all bits
flip each time. As Figure 17.2 shows, A[0]
does flip each time and A[1] flips every other
time or floor(n/2) times for n INCREMENTs.
Similarly, A[2] flips every fourth time or
floor(n/4) times for n INCREMENTs. In general
A[i] flips floor(n/2^i) times in n INCREMENTs
for i = 0, 1, ... , k-1, and A[i] does not
flip for i >= k. So the total number of flips
is:
k-1 infinity
Sum ( floor(n/2^i) ) < n * Sum ( 1/2^i )
i = 0 i = 0
= 2n
by equation A.6 (page 1147). The worst-case
time for n INCREMENT operations on an
initially zero counter is thus O(n). The
average cost, and therefore the amortized cost
per operation is O(n)/n = O(1).
17.2 The accounting method 17.2.1
In the accounting method, we assign differing
charges to different operations, the amount we
charge an operation is its amortized cost. If
an operation's amortized cost exceeds its
actual cost, the difference is assigned to
specific objects in the data structure as
credit. Credit can be used later to pay for
operations whose amortized cost is less than
their actual cost. Note that this differs
from aggregate analysis, in which operations
all have the same amortized cost.
We want to choose the amortized costs so that
the total amortized cost is small so that we
can prove that the average worst case cost is
small. Also, we must choose the amortized
costs so that the total amortized cost of a
sequence of operations is an upper bound for
the actual cost, for any such sequence. If we
denote the actual cost of the i-th operation
by c_i and its amortized cost by c-hat_i, then
we require for all sequences of n operations:
n n
Sum ( c-hat_i ) >= Sum ( c_i ) (17.1)
i = 1 i = 1
The total credit stored in the data structure
is the left side of (17.1) minus the right
side, which should always be non-negative.
Stack operations 17.2.2
Recall that the actual costs of the stack
operations were:
PUSH 1,
POP 1,
MULTIPOP min(k,s),
where k is MULTIPOP's argument and s is the
stack size when it is called. Let us assign
the following O(1) amortized costs:
PUSH 2,
POP 0,
MULTIPOP 0.
We now show that we can pay for any sequence
of stack operations by charging the amortized
costs in dollars. We start with an empty
stack. When we push an item, the actual cost
is $1, and we push the other dollar of the
amortized cost "onto the stack, along with the
item". Thus every item in the stack has a
dollar attached to it.
When we pop an item, we charge the operation
nothing, but pay the actual cost for the pop
with the dollar attached to the popped item.
Moreover, we don't charge MULTIPOP anything
either, since we pay for each pop with the
dollar attached to that item. Since each item
in the stack has a dollar attached to it, the
credit is the number of items/dollars on the
stack, and so is always non-negative. Thus,
for any sequence of n stack operations, the
total amortized cost, O(n), is also a bound
on the total actual cost.
Incrementing a binary counter 17.2.3
We also analyze the binary counter using the
accounting method. The running time of the
counter is proportional to the bits flipped,
which we use as our cost. For this analysis
we charge an amortized cost of $2 to set a bit
to 1, and nothing to set it to 0. When a bit
is set to 1, we pay for it with one dollar and
place the other dollar on that 1-bit, so if we
reset it later we pay for it with that dollar.
In INCREMENT, the cost of resetting bits in
the while loop is paid for by the dollars on
them. At most one bit is set, and therefore
the amortized cost of an INCREMENT is at most
$2. The number of 1's in the counter is never
negative, so the credit is always nonnegative.
Thus for n INCREMENT operations, the total
amortized cost is O(n), which also bounds the
actual cost.
The potential method 17.3.1
Unlike the accounting method in which credit
is attached to data objects, in the potential
method, credit is attached to the entire data
structure. This credit or prepaid work is
called the "potential energy" or "potential",
that can be used to pay for future operations.
We perform n operations on an initial data
structure D_0. For n = 1,2,...,n, we let c_i
be the actual cost of the i-th operation and
D_i be the data structure that results after
applying the first i operations. A potential
function Phi maps each D_i to a real number
Phi(D_i), which is the potential associated
with D_i. The amortized cost c-hat_i of the
i-th operation with respect to Phi is defined
by equation (17.2):
c-hat_i = c_i + Phi(D_i) - Phi( D_(i-1) )
Thus the amortized cost of n operations is:
n
Sum ( c-hat_i ) = (17.3)
i = 1 n
Sum (c_i + Phi(D_i) - Phi(D_(i-1))
i = 1
n
= Sum (c_i) + Phi(D_n) - Phi(D_0)
i = 1
by equation (A.9) for telescoping sums.
If we define Phi so that 17.3.2
Phi(D_n) >= Phi(D_0) then the total amortized
n
cost Sum ( c-hat_i ) is an upper bound on the
i = 1 n
total actual cost Sum ( c_i ).
i = 1
We can guarantee this for all n if we require
that Phi(D_i) >= Phi(D_0) for all i, which
says we always "pay in advance". Usually we
define Phi(D_0) to be 0 (Exercise 17.3-1 shows
what to do if Phi(D_0) > 0).
If the difference Phi(D_i) - Phi( D_(i-1) )
is positive, c-hat_i represents an overcharge
to the i-th operation; and if the difference
is negative, the amortized cost c-hat_i
represents an undercharge and the actual cost
is paid by the decrease in the potential.
The amortized costs above depend on how Phi
is chosen. Different functions Phi can give
different amortized costs. There are often
trade-offs in choosing Phi; the best choice of
Phi depends on the desired time bounds.
Stack operations
For our stack example, we define Phi to be
the number of items on the stack. So for the
starting empty stack D_0, Phi(D_0) = 0. Since
the number of items on the stack is never
negative, D_i has nonnegative potential, so
Phi(D_i) >= 0 = Phi(D_0)
17.3.3
Thus the total amortized cost of n operations
represents an upper bound on the actual cost.
Now we compute the amortized cost of the
stack operations. If the i-th operation on a
stack containing s items is a PUSH, then the
potential difference is:
Phi(D_i) - Phi( D_(i-1) ) = (s + 1) - s = 1
So by equation (17.2), the amortized cost is:
c-hat_i = c_i + Phi(D_i) - Phi( D_(i-1) )
= 1 + 1 = 2
Now suppose that the i-th operation is a
MULTIPOP(S,k) and that k' = min(s,k) items are
popped off the stack. The actual cost of the
operation is k', and the potential difference
is: Phi(D_i) - Phi( D_(i-1) ) = -k'.
Thus the amortized cost of the MULTIPOP is:
c-hat_i = c_i + Phi(D_i) - Phi( D_(i-1) )
= k' + ( -k' ) = 0
Similarly, the amortized cost of POP is 0 too.
The amortized cost of the three operations is
O(1), so the total amortized cost of n
operations is O(n). Since we have argued that
Phi(D_i) >= Phi(D_0), the total amortized cost
is an upper bound for the total actual cost.
The worst-case actual cost of n operations is
therefore O(n) also.
Incrementing a binary counter 17.3.4
For the binary counter, we define the
potential to be b_i, the number of 1-bits in
the counter after the i-th operation.
Let us compute the amortized cost of an
INCREMENT operation. Suppose that the i-th
INCREMENT resets t_i bits. The actual cost of
the operation is then at most t_i + 1, since
at most one bit is set to 1. If b_i = 0, then
the i-th operation reset all k bits, and so
b_(i-1) = t_i = k. If b_i > 0, then we have
b_i = b_(i-1) - t_i + 1. In either case,
b_i <= b_(i-1) - t_i + 1, and the potential
difference is:
Phi(D_i) - Phi( D_(i-1) )
<= ( b_(i-1) - t_i + 1 ) - b_(i-1)
= 1 - t_i
The amortized cost is therefore:
c-hat_i = c_i + Phi(D_i) - Phi( D_(i-1) )
<= (t_i + 1) + ( 1 - t_i ) = 2
If the counter starts at zero, Phi(D_0) = 0.
Since Phi(D_i) >= Phi(D_0), the total
amortized cost of n INCREMENT operations is an
upper bound on the total actual cost, and so
the worst-case cost of n INCREMENT operations
is O(n).
17.3.5
The potential method also gives us a way to
analyze the counter even when it doesn't start
at zero. There are initially b_0 1's and
after n INCREMENT operations there are b_n 1's
where 0 <= b_0, b_n <= k, the number of bits
in the counter. We can rewrite (17.3) as:
n n
Sum(c_i) = Sum(c-hat_i) - Phi(D_n) + Phi(D_0)
i = 1 i = 1
We have c-hat_i <= 2, and since Phi(D_0) = b_0
and Phi(D_n) = b_n, the total actual cost is:
n n
Sum(c_i) <= Sum( 2 ) - b_n + b_0
i = 1 i = 1
= 2n - b_n + b_0
In particular, note that since b_0 <= k, as
long as k = O(n), the total actual cost is
O(n). In other words, if we execute at least
n = Omega(k) INCREMENT operations, the total
actual cost is O(n), no matter what the
initial value of the counter was.