CS 8761 Natural Language Processing - Fall 2004 - Automated Essay Grading

Final Project - Alpha Stage, due Mon Nov 15, before noon

This may be revised in response to your questions. Last update Weds Nov 3, 2 pm


To design, implement and deploy an alpha version of your essay grading system.

This system should be web based, and include baseline functionality for two of the following four components to your system: You will submit your system such that I can install and use it, and you will also have a web site up and running that can be used. It is very likely that I will demo your systems to the class on Mon, Nov 15.


Your system should have a simple web interface that provides the user with a prompt (i.e, question) to answer. This question should be selected from a file of possible prompts that you provide. The user should enter their essay, and then submit it for grading. The system should respond relatively quickly and be interactive. At this point do not try and provide an overall score, just provide feedback from the above components.

Your system should be submitted "pre-trained", so that I can begin using it immediately. In other words, if you do any calculations based on large corpora, you should have already done those computations and provide me with a data file that includes those results. Of course there may be some dynamic computation required, and that is fine. For example, you may look up information on the Web based on the users input. However, if you use an LSA style approach, please create the co-occurrence matrix ahead of time, and include that in your distribution.

It is up to your team to decide how you will approach each of these problems. You may certainly use ideas from the published literature, just make sure to acknowledge this in your documentation (described below) by providing citations and references.


The installation of your system should simple, and follow the standard "3 step Perl" install. These steps consist of the following:
perl Makefile.PL PREFIX=/home/cs/tpederse etc.
make test
Note that if you rely on existing tools (like the Brill Part of Speech Tagger, WordNet, etc.) you can simply provide me with detailed instructions about where I can find those tools, and how to install them. Do not assume that I know how to install those, and do not assume that I already have them available. Also, do not simply refer me to the instructions in that package, please provide a concise set of instructions that includes any and all tools that might need to be available.

Note that the PREFIX variable indicates that I will not have supervisor access, and I will install into my own personal directory. It is fine if there are other directives required for Makefile.PL, simply indicate what they are in your INSTALL file.

Your distribution should include a plain text file named INSTALL the provides detailed instructions on how to install your system. You should assume that the user of your system only has this documentation available, and is not an expert in system administration. Thus, if any other tools need to be installe, paths need to be set, Makefile.PL variables need to be set, etc. please provide detailed instructions about how to do that. Assume that your user does not have supervisor access!


Your distribution should include a file named README.pod that is written in perldoc that describes your overall system.

You should assume that the user has no specialized knowledge of automated essay grading. Thus, your README.pod should begin with a general introduction to the problem of automated essay grading. This should include a brief description of the history and related work in this area, and it should be written in the style and form of a related work section in a thesis or thesis proposal.

You should then introduce each of the four problems described above. In other words, do not assume that the user will know what gibberish detection, or relevance measurement is. Provide examples of text that would "trigger" your system to provide guidance to the essay writer. You should do this for all four systems, even though you only need to have baseline implementations of two of them.

Then, you should describe the specific approach that you are taking for your two baseline solutions, and also describe your overall plan for each other components. You should clearly indicate what possible approaches you are considering for each component, and also describe who is going to do what.

Finally, compare and contrast your proposed approach with existing techniques, and clearly credit any publications, systems, etc. that might have given you ideas.

To summarize, your README.pod should consist of the following sections:

Submission Guidelines

You should package your system as a single compressed tar file that is named as TEAMNAME-0.01.tar.gz. This should include all of the code needed to run your system, and the README and INSTALL documentation described above. If you are using CPAN modules in your system, make sure those dependencies are described in your Makefile.PL, and I will obtain those modules via CPAN. The same is true of other outside packages. If they are not available via CPAN (such as the Brill Tagger, WordNet) provide detailed instructions as to where I can find those and how to install them. You do not need to include those in your distribution.

Your system should unpack into a directory named as TEAMNAME-0.01. TEAMNAME should be your assigned team name, written in all capital letters. If there are embedded spaces in your name (e.g., BOCA JUNIORS) you should replace the space with a -, to result in the name (e.g., BOCA-JUNIORS-0.01).

If you are using sourceforge, upload your system and I will download from there. Please note that sourceforge provides a time stamp when you upload. In your README, please prominently provide a URL where I can run your system. Please note that I will also install it on my own system as well.

Finally, make sure that you specifically address the issue of "terms of use". In the end, we will be posting your code on the Web, and you should specifically address the issue of how the code can be used and distributed. One option is the GNU CopyLeft. You are free to investigate other licensing terms, but you must specifically address this issue and follow the accepted standards for including licensing information in your code and documentation. Also make sure to clearly identify the authors of individual programs, and provide contact information. If one person does all the work for a particular component, then their name should be the only one on that particular component. In cases where the effort is shared, then all team members should be credited.


This is a team assignment. You are strongly advised to divide up the work of the project into tasks that can be carried out in parallel by various team members. All team members should be acknowledged in the comments, etc. and all teammates will receive the same grade. Do not work with other teams, and do not discuss your approach with other teams. Each team should operate independent of all the other teams. Make your own decisions as a team and do not be influenced by the decisions of other teams if you happen to hear of them accidentally.

by: Ted Pedersen - tpederse@umn.edu