CS 5761 - Introduction to Natural Language Processing
Programming Assignment 2 - Submit via
web drop by 5pm Monday Feb 16.
To develop techniques that will be useful in creating profiles of written
documents. To gain experience with Perl file handling and hashes.
Write a Perl program called profiler.pl that will take as input a rank
cutoff (described below), a split value (described below) and an arbitrary
number of text files (1 or more). If multiple files are given as input,
treat all of them as one large document. You should convert all text to
upper or lower case, and remove or ignore punctuation. Your program
should output the following information in a nicely designed report.
For example, the following command will run the profiler and display the
top 30 ranked words, and display the word types that are unique to the
last 5 percent of the text. The text to be processed is found in two
files (holmes1.txt and bible.txt) but will be treated as one big file.
profiler.pl 30 5 olivertwist.txt davidcopperfield.txt
There will be some experiments for you to run and report on as a part of
this assignment. That will be discussed in more detail in the lecture, and
will be due a few days after the code is to be turned in.
You should conduct the following experiments, and prepare a short
presentation to be given in the lab on Thu Feb 19. This
will count for 30% of the grade on this assignment.
You should select two corpora that are from different genres. Each should
consist of at least 100,000 tokens. For example, you might choose a novel
as your first corpus, and news wire text as your second corpus. Note that
your two corpora should be fairly distinct, so do not select two different
novels or text from two different newspapers. You should be able
to find suitable text by searching around the Internet. Don't
forget the links that are found on the class web page.
Run your profiler.pl program on each corpus with two different settings
for S, 50 and 5. Interpret what these results tell you about the nature of
the corpora, and about the nature of Zipf's Law.
You should prepare a presentation to give to the class during the lab.
This should consist of powerpoint slides, or handouts. In either case,
submit your slides or handouts to the webdrop by 4pm on Thursday, and I
will put the slides on the computer in HH 302 for projection, or I will
print out enough copies of your handouts so that each member of the class
Your presentation should summarize the characteristics of
your corpora, the results of each of the 4 possible runs of profiler, and
then your conclusions about what this tells us about the nature of the
text and Zipf's Law. Note that you must draw conclusions - you can not
simply summarize your results.
Policies (see syllabus for more details)
Please comment your code. In particular, note where each piece of data
requested above is being collected in your code. Also make sure you name,
class, etc. is clearly included in the comments.
It is fine to use a Perl reference book for examples of loops,
variables, etc., but your profiler specific code must be your own, and not
taken from any other source (human, published, on the web, etc.)