Ted Pedersen - CS 8995 Corpus Based Natural Language Processing

CS 8995 Corpus Based Natural Language Processing

UNDER CONSTRUCTION. THIS MAY BE REVISED IN RESPONSE TO QUESTIONS. LAST UPDATE Tue APRIL 3, 10 am

Final Project: Empirical Methods for Multilingual Text

Stage 2 - Due Mon, Apr 16, 4 pm

Objectives

To refine the methods developed for sentence alignment of parallel corpora in the previous assignment.

Specification

Your team should further study and improve the performance of the alignment algorithm that you implemented in stage 1. Feel free to use whichever team member's stage 1 resources appear to be the most effective. You may assume that one of the languages to be aligned is English.

For all code submitted, please include a header comment that includes the name of your team and the team members, as well as brief instructions on how to use the program. Also, if you have used code from a previous team, please give full and accurate credit to the original author/s and clearly explain how you have changed their code. You should also explicitly state that you have received the permission of the original authors to use this code.

Sentence Boundary Detection
One limitation of the alignment programs in the previous stage was that they usually required sentence boundary detection algorithms for both languages, and as such required knowledge of the rules of punctuation in both languages to succeed. In this stage, assume that you know the rules of boundary detection in English, but not in the other language. Thus, your sentence boundary detection algorithm should function exactly the same with English and French as it does with English and Chinese.

In general I would prefer that you avoid language specific boundary detection algorithms (except for English). An alignment program such as Gale-Church's is relatively language independent. Ideally we'd like to keep the entire process language independent so as to maximize the portability and usefulness of our work here.

Unicode might provide some help in maintaining language independence. The character classes
\p{IsWord} and \p{IsPunct}
will identify word characters and punctuation marks for the language you are processing. Being able to differentiate punctuation from word characters is at least a start, and may be an improvement on relying strictly on length based criteria for boundary detection.

From a software engineering point of view, it probably makes sense to separate your boundary detection component from Gale-Church. This will make it easier for you to test alternatives and also reduce the risk of bugs creeping into Gale-Church. If you do write a separate boundary detection program, simply name it teamname-bound.pl and submit it as described in the "Other Information" section.
Baseline Alignment Algorithm
Implement the following simple baseline alignment algorithm: Locate sentence boundaries using your new detection algorithm, and then simply assume that sentence number n in English aligns with sentence n in the other language. Evaluate this baseline algorithm relative to the Gale-Church alignment algorithm. Make sure that your evaluation criteria is functioning as described below.
Evaluation Criteria
Your evaluation criteria must account for partial matches in the sentence alignments. In other words, assume that sentence boundary detection will not be perfect, and that you may sometimes have alignments that are partially correct. Your evaluation program should check the text in the sentences and not simply rely upon the sentence numbers reported. For example, suppose we are scoring the alignment of the following pair:

Our alignment program produces the following:
```
< alignment 1=1>                          < alignment 1=1>
This is my English sentence.             Esto es
< /alignment>                             < /alignment> 
```
The Gold Standard:
```
< alignment 1=1>                          < alignment 1=1>
This is my English sentence.               Esto es mi oracion Ingles.
< /alignment>                             < /alignment> 
```
If the evaluation criteria was based strictly on line numbers (1=1) this would be considered a perfect alignment. Clearly it is neither perfect nor totally incorrect and your scoring mechanism should reflect that.

A second pitfall of line-number based evaluation is as follows. Suppose our alignment program produces the following:
```
< alignment 1=2>                          < alignment 2=1>
This is my English sentence.             Esto es mi oraction Ingles. 
< /alignment>                             < /alignment> 
```
The Gold Standard:
```
< alignment 1=1>                          < alignment 1=1>
This is my English sentence.               Esto es mi oracion Ingles.
< /alignment>                             < /alignment> 
```
If we simply check alignment numbers (1=1) then this alignment will be judged as incorrect. In fact it is completely correct and should be scored as such. Misnumberings of sentences should not invalidate good alignments.

Instead of reporting a single F-measure score, your evaluation program should report precision, recall, and accuracy. I will post the evaluation programs soon after the deadline. You may not revise your submissions after the deadline, so please make sure you are submitting a working version of your evaluation program. Also, make certain that your evaluation program only outputs three values in a single line and in the following order: precision, recall, accuracy. These values should be delimited by spaces.
Written Analysis of Results
Summarize the results obtained with your baseline alignment program and the Gale-Church alignment. Comment on the following:
- Which method was more accurate? How great was the difference in accuracy between the baseline and gale-church? How do you account for this?
- To what extent did the degree of reordering in the translation affect the accuracy of your method? Why is affected as much (or as little) as it is? How did you assess this? (HINT: consider reordering one of the existing gold standard data sets to make the alignment problem more difficult.)
- Did results vary depending on the languages involved? Was your method substantially more accuracy for one language over another? Why?
- What was the impact of sentence boundary detection on the accuracy of your alignment algorithms (both baseline and gale-church)?
- Is your gale-church implementation correct? How do you know? (HINT: consider comparing the results of your alignment program with one of the standard C implementations that are available.)
You are encouraged to add any other comments or observations that you might have. The above points should be considered a minimum. The report ARCADE report at: http://www.lpl.univ-aix.fr/projects/arcade/report1-en/index.html provides a nice example of how a comparative evaluation of alignment programs can be discussed. (There is a link to this on our textbooks web site.)

Evaluation Results

THESE WILL BE DUE ON TUE, APRIL 17 at MIDNIGHT.

Run your baseline algorithm and the gale-church algorithm with all 7 gold standard data sets. Evaluate these results with all 7 evaluation criteria. Follow the procedure from stage 1, where sentence alignment information is stripped from the gold standard data prior to running your alignment program. Compare the results of your alignment program with the gold standard alignment using the evaluation criteria. Report your results in the following format. This example is for team MORELIA using the CRUZAZUL evaluation criteria with both the baseline (-B) and Gale-Church (-CG) alignment programs. Your results should include 7*14=98 such entries.

MORELIA-B  CRUZAZUL CRUZAZUL precision recall accuracy
MORELIA-B  MORELIA CRUZAZUL precision recall accuracy
MORELIA-B  TOLUCA CRUZAZUL precision recall accuracy 
MORELIA-B  UNAM CRUZAZUL precision recall accuracy 
MORELIA-B  PACHUCA CRUZAZUL precision recall accuracy 
MORELIA-B  TIGRES CRUZAZUL precision recall accuracy 
MORELIA-B  PUEBLA CRUZAZUL precision recall accuracy 
MORELIA-GC CRUZAZUL CRUZAZUL precision recall accuracy
MORELIA-GC MORELIA CRUZAZUL precision recall accuracy
MORELIA-GC TOLUCA CRUZAZUL precision recall accuracy 
MORELIA-GC UNAM CRUZAZUL precision recall accuracy 
MORELIA-GC PACHUCA CRUZAZUL precision recall accuracy 
MORELIA-GC TIGRES CRUZAZUL precision recall accuracy 
MORELIA-GC PUEBLA CRUZAZUL precision recall accuracy

Other information

There are five items to submit. You may also submit a boundary detection program if you have separated that from Gale-Church. This is optional.

alignment program revised for sentence boundary detection described above, and for any other fixes improvements you may have elected to include.
baseline algorithm
revised evaluation program
written analysis
evaluation results (not due until midnight Tue Apr 17)

Please use turnin to submit all five items. Remember that you can only use turnin from hh33812. No email submission is necessary. The proper turnin commands are:

turnin -c cs8995 -p p2a teamname-gc.pl (gale-church sentence alignment)
turnin -c cs8995 -p p2b teamname-b.pl (baseline algorithm)
turnin -c cs8995 -p p2c teamname.(pdf|ps) (written analysis)
turnin -c cs8995 -p p2d teamname-eval.pl (evaluation program)
turnin -c cs8995 -p p2e teamname-results.dat (evaluation results)
turnin -c cs8995 -p p2f teamname-bound.pl (optional separate boundary detector)

This is a team project. Please consult with and work with your team members closely. You may divide the work as you see fit, and you have considerable discretion in your approach to this problem.

Please note that the deadline will be enforced by automatic means. Any submissions after the deadline will not be received.

by: Ted Pedersen - tpederse@d.umn.edu

CS 8995 Corpus Based Natural Language Processing

UNDER CONSTRUCTION. THIS MAY BE REVISED IN RESPONSE TO QUESTIONS. LAST UPDATE Tue APRIL 3, 10 am

Final Project: Empirical Methods for Multilingual Text

Stage 2 - Due Mon, Apr 16, 4 pm

Objectives

Specification

Sentence Boundary Detection

Baseline Alignment Algorithm

Evaluation Criteria

Written Analysis of Results

Evaluation Results

Other information