Part I Foundations I.1 Outline: Chapter 1: Gives an overview of algorithms and their role in computing. The term algorithm is defined and a case is made that algorithms are a kind of technology. Chapter 2: Gives examples of algorithms and a pseudocode used to express them. These are examples of the incremental and divide- and-conquer techniques for algorithm design. Chapter 3: Defines "asymptotic notation", which is used to measure running times and memory requirements of algorithms. Chapter 4: Shows how to solve recurrences that usually arise from run-time analysis of recursive algorithms, and divide-and-conquer in particular. Chapter 5: Treats probabilistic analysis and randomized algorithms. Probabilistic analysis is useful in analyzing average-case performance of algorithms. An algorithm is said to be randomized if its behavior is determined partly by using a random-number generator -- to ensure average-case performance for example. Chapter 1 The Role of Algorithms in 1.1.1 Computing 1.1 Algorithms Informal definition: an algorithm is any well- defined (computational) procedure that takes a value or set of values as input and produces a value or set of values as output. Thus an algorithm is a sequence of (computational) steps that transform the input into the output. Also we can view an algorithm as a tool for solving a computational problem, specified in terms of its input/output relationship. Example: the sorting problem: Input: a sequence of n numbers. Output: a permutation of the input with a_1' <= a_2' <= ... <= a_n'. A particular input is called an instance of a computational problem. For example, the input sequence <31,41,59,26,41,58> is an instance of the sorting problem. Definition: we say an algorithm is correct if it halts with correct output for every input instance. What kinds of problems are solved by 1.1.2 algorithms? Examples of "real-world" problems solved by algorithms: - The Human Genome Project: identifying all 100,000 genes in human DNA, determining the sequences of the 3 billion chemical base pairs the make up the DNA, storing this information in databases, and developing tools for analysis of this data. - On the Internet: routing data and quickly locating web pages with a search engine. - Electronic commerce: using secure passing of critical information such as credit card numbers, passwords, etc. - In manufacturing and business: allocating resources in the most beneficial way. This can often by solved by linear programming. Examples of problems solved in this book: - Given a road map marked with distances between cities, find the shortest route from one city to another -- solved using graph algorithms in Chapter 24. - Solving integer equations ax = b (mod n), which is used in cryptography (Chapter 31). 1.1.3 - Given a sequence of n matrices of different (compatible) sizes, find their product A_1xA_2x...xA_n. Since matrix multiplication is associative, this can be done in many ways. Dynamic programming (Chapter 15) shows how to do the matrix multiplications with the fewest element-wise multiplications. - Given n points in the plane, find its convex hull, the smallest polygon containing the points (Chapter 33). Two characteristics of interesting algorithms: 1. There are many possible algorithms that solve the problem, most of which are not what we want. 2. There are practical, i.e. "real-world" applications. Data structures Definition: A data structure is a way to store and organize data so as to facilitate access and modifications. No one data structure is best for all situations, so it is important to know strengths and weaknesses of a selection of data structures. Technique 1.1.4 Although many algorithms have been published, you may encounter a problem for which you can't find a solution. In that case, you must design and analyze an algorithm yourself -- this book presents such design and analysis techniques. Hard problems We are interested in efficient algorithms - how long the algorithm takes to compute its result. Some problems have no known efficient solution. A subset of these problems are the NP-complete ones (Chapter 34). Why are NP-complete problems interesting? 1) No one has proved that an efficient algorithm cannot exist. 2) The set of NP-complete problems has the property that if there is an efficient algorithm for any one of them, then efficient algorithms exist for all of them. 3) Several NP-complete problems are similar to problems with efficient solutions - a small change in the problem can result in a big change in the efficiency of its solution. 1.1.5 It is useful to know about NP-complete problems since they often show up in real-life applications. Probably the best known example is the "traveling-salesman problem". Rather than attempting to find an efficient solution for such a problem (which is unlikely), it is better to come up with an approximate solution efficiently. Chapter 35 discusses such "approximate algorithms". Also, it is useful to know about problems that have been proved to have no solutions. Two examples are the halting problem (proved insolvable by Alan Turing), and the word problem for groups (proved insolvable by Petr Sergeevich Novikov in 1953). 1.2 Algorithms as a technology 1.2.1 Even if computers were infinitely fast and memory were free, it would still be important to study algorithms since we would still want to know that they terminated - and with the correct answer. Then you would probably use the easiest algorithm to implement. Of course computers are not infinitely fast, nor is memory free, so it is important that algorithms be efficient in time and space. Efficiency As an example of comparative efficiency, we consider insertion sort and merge sort - both used to sort n items. Insertion sort takes time roughly equal to c1 x n^2, where c1 is a constant (independent of n). Merge sort takes time roughly equal to c2 x n x lg(n), where lg(n) = log_2(n) and c2 is a constant (which is usually larger than c1). As a concrete example, we assume insertion sort has been programmed in machine language by a good programmer and runs on computer A, which executes a billion instructions per second. And merge sort has been programmed by a so-so programmer in some high-level language and runs on computer B, which can execute ten million instructions per second. 1.2.2 Taking this into account, assume that insertion sort takes 2n^2 instructions to sort n numbers, and merge sort takes 50n x lg(n). To sort one million numbers, computer A with insertion sort takes 2 x (10^6)^2 instructions ------------------------- = 2000 seconds 10^9 instructions/second and computer B with merge sort takes: 50 x 10^6 lg(10^6) instructions ------------------------------- = about 99.66 10^7 instructions/second seconds To sort 10 million numbers, insertion sort takes about 2.3 days, and merge sort takes about 19.4 minutes -- about 170 times faster. Algorithms and other technologies The above example shows that algorithms are just as important as other technologies when evaluating system performance. 1.2.3 How important are algorithms when compared to other advanced computer technologies such as: 1) hardware with high clock rates, pipelining, and superscalar architectures, 2) easy-to-use GUIs, 3) object-oriented systems, and 4) local-area and wide-area networking? Both GUIs (Graphical User Interfaces) and especially hardware are designed by using algorithms. Object-oriented software is created with compilers, interpreters, and assemblers, all of which make heavy use of algorithms. Heavy use is made of algorithms also for both designing and routing data around networks. Also, as hardware becomes faster, it becomes possible to solve problems that were too big before. But these solutions can only be found by using the most efficient algorithms. According to the authors of the text: "A solid base of algorithmic knowledge and technique is one characteristic that separates the truly skilled programmers from the novices."