CS 5761 - Introduction to Natural Language Processing
Programming Assignment 4 - Submit via
web drop by 5pm Friday March 26.
To gain experience learning n-gram language models from text.
Design and implement a Perl program called ngram.pl that will learn an
n-gram language model from a given body of text. Your program should then
generate a given number of sentences based on that n-gram model. See the
discussion on pages 202-206 of your text for further details.
Your program should work for any value of n, and should output m
sentences. Convert all text to lower case, and make sure to include
punctuation in the n-gram models. Your program should learn a single
n-gram model from any number of input files.
Your program should run as follows:
ngram.pl n m input-file/s
so running your program like this:
ngram.pl 3 10 book1.text book2.text
should result in 10 randomly generated sentences based on a tri-gram
model learned from book1.text and book2.text.
Make sure that you separate punctuation marks from text and treat them as
tokens. Also treat numeric data as tokens. So, in a sentence like
my, oh my, i wish i had 100 dollars .
you should have 12 tokens
my , oh my , i wish i had 100 dollars .
Your program will need to identify sentence boundaries, and your ngrams
should *not* cross these boundaries. For example, you could have input
He went down the stairs
and then out the side door.
My mother and brother
You should treat this as two sentences, as in:
He went down the stairs and then out the side door .
My mother and brother followed him .
To identify sentence boundaries, you may assume that any period,
question mark, or exclamation point represents the end of a sentence.
(In general this assumption is wrong, but is perfectly adequate for our
purposes here.) When generating a sentence, keep going until you find a
terminating punctuation mark. Once you observe that then the sentence is
If the length of a sentence in the input text file is less than n, then
you may simply discard that sentence and not use it when computing
Policies (see syllabus for more details)
Please comment your code. You must provided a detailed description of your
spelling correction algorithm in your source code comments. This should
focus on how you score the candidate corrections for a word. Also make
sure you name, class, etc. is clearly included in the comments.
It is fine to use a Perl reference book to provide examples of loops,
variables, etc., but your ngram.pl specific code must be your own, and
not taken from any other source (human, published, on the web, etc.)