# This is the pipeline used to annotate Gigaword English v.5

# (Annotated Gigaword, Napoles et al. 2012).

#

# Courtney Napoles, cdnapoles@gmail.com

# 2012-07-03

NOTES

See pipeline.sh for the full pipeline and usage of individual steps

If you are running a copy of this, you need to modify scripts/

splitta.1.03/sbd.py so that the paths for SVM_LEARN and SVM_CLASSIFY

point to your installation.

Note that the pipeline uses a parallel environment (8 threads for

parsing) so please set your configurations accordingly

(qsub -l num_proc=8,mem_free=16G,h_vmem=22G).

Be sure to set the environment encoding to UTF-8.

USAGE

To run:

./pipeline file_to_annotate.xml working_directory [OPTIONS]

To just run the annotators:

java -Xmx16g -cp bin:lib/stanford-corenlp-2012-05-22.jar:lib/my-xom.jar:lib/stanford-corenlp-2012-05-22-models.jar:lib/joda-time.jar \

edu.jhu.annotation.GigawordAnnotator --in <TESTFILE>

If you'd like to annotate a file that contains a single document

without any SGML markup, add "--sgml f". However, for annotating a

large quantity of files this is unadvisable, because loading the

Stanford models takes a couple of minutes. It is more efficient to

include several documents in one file (and documents should be

formatted like <DOC><TEXT>parses</TEXT></DOC>).

FILE FORMAT

sample.txt contains a sample file format. If using SGML markup

(which is recommended because then multiple documents can be stored

in the same file), the following format is assumed:

<DOC id="xx">

<TEXT>

...

</TEXT>

</DOC>

<DOC ...

Any tags in between <DOC> and <TEXT> are ignored but passed through

intact. The only tag allowed in <TEXT> is <P>. All text in the

<TEXT> element will be processed and annotated. The pipeline assumes

that each line is EITHER sgml markup or text (so do not put a tag

on the same line as text. The pipeline does not detect/correct

invalid SGML but it will convert SGML to XML (by adding a root

element and escaping <, >, and &.

DEPENDENCIES

Software versions used:

Splitta 1.03

Stanford CoreNLP 1.3.2

Requirements:

jgrapht.jar

joda-time.jar

my-xom.jar

splitta.1.03

stanford-corenlp-2012-05-22.jar

stanford-corenlp-2012-05-22-models.jar

svm_light.6.02

umd-parser.jar

wsj-6.pml # grammar file