CS 8761 Natural Language Processing - Fall 2004
Assignment 2 - Due Friday Oct 15, 2004, noon
This may be revised in response to your questions.
Last update Sunday Oct 10, 4pm
To estimate the entropy of English and interpret those results based on
experiments using literary and newswire text. You may also gain an
appreciation for the difficulties of sentence boundary detection.
Write a Perl program called shannon.pl that will play the "Shannon game".
Your program should accept as input an arbitrary number of sentences that
are read from STDIN. The user must then guess each sentence, letter by
letter. You should estimate the entropy of English based on these guesses.
Your Shannon Game program should display one sentence at time, where all
the positions in the sentence have been covered by "*" characters. When
the user guesses the correct letter it should be displayed in place of the
"*". When the user guesses the complete sentence, the total entropy for
that sentence should be displayed, as should a running total that
reflects the entropy across all the sentences processed thus far.
You should not assume that sentence boundaries have been determined in
your input text. Thus, you should provide a sentence boundary detector
program called boundary.pl to create the input to your Shannon game
program. You should require that each sentence selected by boundary.pl
has a minimum of 7 words. Make sure that once a sentence is
selected that it is not selected again during the same run of the program.
Your sentence boundary detection program should also convert the
text to upper case and remove all punctuation and non-alpha characters
except for spaces. Please note that sentence boundary detection is a
difficult problem in its own right, so you should not be surprised if
you have an imperfect solution. That is ok, as long as you are able to
get the simple cases correct. At a minimum, your boundary detector should
identify sentences that end with . ? or ! and have no internal uses of
these characters (like might occur in an abbreviation or quote). You
should be able to identify a sentence that occurs on multiple lines, or
multiple sentences per line. For example, the following are cases you
should be able to handle.
This sentences is
going to go on and on for a
long while, and then end.
What a great sentence. This one is even better!
My friends, I wish that I would be with you today, but alas, I can't!
Examples of cases that I wouldn't necessarily expect you to handle (but
would be happy if you did) are as follows:
Dr. Johnson, the noted Ph.D. in anthropology, is a great person.
My friend said, "Hey! Ted! You're lost!" and I had to agree.
These two programs should be run as follows from the command line:
boundary.pl 10 chapter1.txt chapter2.txt | shannon.pl outfile1 outfile2 outfile3 outfile4
boundary.pl will select 10 random sentences from chapter1.txt and
chapter2.txt and output them to STDOUT. shannon.pl will read from STDIN
and display the sentences one by one to the user (where the actual
letters and spaces have been covered with "*" characters). outfile1
outfile2, outfile3, and outfile4 (whose names are specified by the user)
are output files that are described
If the user happens to guess the same letter twice for the same position,
don't count this more than once. Please keep track of the letters guessed
for each position to avoid double counting.
You may assume that the command line arguments are valid. There is no need
to include command line error checking. Please note that I will test your
program automatically on a csdev machine so please test it on the same
and follow exactly same command line format.
Your program should reveal the sentence as letters are guessed. This does
not need to be a fancy display - simple ascii output is fine.
After each sentence is guessed, your program should output the entropy
associated with that sentence, and a cumulative total of entropy for all
sentences processed thus far. You only need display two digits of
precision to the right of the decimal.
Your program should also create three output files that can be named from
the command line. Above they are called outfile1, outfile2, outfile3, and
outfile4, and will be referred to as such here, however you should assume
that the user can name them anything they like.
outfile1: A table of the average number of guesses before the given
letter was guessed correctly.
This shows that when A was the correct letter, it took 4.44 tries to guess
it correctly. Your table should have 27 entries, one for each of the 26
letters and also for space.
outfile2: A table showing the average number of guesses to get a letter
correct given that the previous letter is as shown:
This shows that when A was the letter preceding the letter to be guessed,
it took 3.99 turns to guess that next letter correctly.
outfile3: A table showing the average number of guesses to get a letter
correct given that the following letter is as shown:
This shows that when A follows the letter to be guessed, it takes 2.99 to
get that preceding letter correct (on average).
The values in these files should be computed to 2 digits of precision to
the right of the decimal point.
outfile4: A table that displays each sentence that was guessed, the
number of guesses per letter, and the entropy for that sentence and
overall. Each entry in this table should be formatted as follows:
M Y S E N T E N C E I S H E R E T O D A Y
10 2 1 9 3 2 1 1 1 1 1 1 8 3 1 10 2 1 1 1 9 1 1 1 1
Sentence Entropy: X.XXXX
Cumulative Entropy: X.XXXX
Note that the sentence above only has 5 words, this is just to show you
an example of the output format for outfile4. Remember that each of your
sentences should contain at least 7 words.
Experiments and Report
You should produce a written report describing the outcome of the
following experiments. This should be a plain text file called
experiments.txt. This report should begin with a description of your
method for computing entropy of English.
Download a novel from
(www.gutenberg.net). In your report, please name the novel and also
provide a specific URL where it can be directly downloaded from.
Run the Shannon Game for
20 randomly selected sentences from this novel. Examine the results in
outfile1, outfile2, outfile3, and outfile4. What conclusions can you draw
from these results? Please
make an effort to incorporate all the different pieces of information you
have available into your analysis. Make sure you include your input
and output files as well as the entropy values in your report.
Now use the apw.100000 file to select 20 random sentences and repeat the
experiment and analysis that you conducted for the novel.
After running experiments on both kinds of texts, what general conclusions
can you draw about the entropy of English? Make specific references to the
results in your output files, as well as the overall entropy values.
Make sure to include your view on whether or not the entropy of
English is different depending on the type of text. Make sure to support
your position with evidence from your experiments.
I am not looking for a particular answer. There are many conclusions one
can draw from this kind of data. I'm curious as to what you find
interesting, and want to see how you analyze this type of data.
If your analysis suggests that other experiments are appropriate, feel
free to carry them out and comment on them. Make sure you do my original
set of experiments above however, even if you introduce your own.
Please name your programs 'boundary.pl' and 'shannon.pl'. Please name your
report 'experiments.txt'. Each of these should be plain text files. Make
sure that your name, date, and class information are contained in each
file, and that your programs are commented using perldoc.
Place these three files into a directory that is named with your umd user
id. In my case the directory would be called tpederse, for example. Then
create a tar file that includes this directory and your three files.
Compress that tar file and submit it via the web drop from the class home
page. Please note that any submissions after the deadline will be not be
This is an individual assignment. You must write *all* of your code on
your own. Do not get code from your colleagues, the internet, etc. You are
welcome to discuss the Shannon game in general with your classmates,
friends, family, etc., but do not discuss implementation details. You
must also write your report on your own. Please do not discuss your
interpretations of these results amongst yourselves. This is meant to
make you think for yourself and arrive at your own conclusions.