HOW-TO GUIDE: Installing and running the Joshua Decoder

by Chris Callison-Burch (Released: June 12, 2009)

This document gives instructions on how to install and use the Joshua decoder. Joshua is an open-source decoder for parsing-based machine translation. Joshua uses the synchronous context free grammar (SCFG) formalism in its approach to statistical machine translation, and the software implements the algorithms that underly the approach.

1. Install the software
2. Prepare your data
3. Create word alignments
4. Train a language model
5. Extract a translation grammar
6. Run minimum error rate training
7. Decode a test set
8. Recase the translations
9. Score the translations

If you use Joshua in your work, please cite this paper:

Zhifei Li, Chris Callison-Burch, Chris Dyer, Juri Ganitkevitch, Sanjeev Khudanpur, Lane Schwartz, Wren Thornton, Jonathan Weese and Omar Zaidan, 2009. Joshua: An Open Source Toolkit for Parsing-based Machine Translation. In Proceedings of the Workshop on Statistical Machine Translation (WMT09). [pdf] [bib]

These instructions apply to Release 1.3 of Joshua, which is described in our WMT09 paper. You can also get the latest version of the Joshua software from the repository with the command:

svn checkout https://joshua.svn.sf.net/svnroot/joshua/trunk joshua

• Apache Ant - ant is a tool for compiling Java code which has similar functionality to make.
• Swig - swig is a tool that connects programs written C++ with Java.

Before installing these, you can check whether they're already on your system by typing which ant and which swig.

In addition to these software development tools, you will also need to download:

• The SRI language modeling toolkit - srilm is a widely used toolkit for building n-gram language models, which are an important component in the translation process.
• The Berkeley Aligner - this software is used to align words across sentence pairs in a bilingual parallel corpus. Word alignment takes place before extracting an SCFG.

After you have downloaded the srilm tar file, type the following commands to install it:

mkdir srilm 
mv srilm.tgz srilm/ 
cd srilm/ 
tar xfz srilm.tgz 
make 

If the build fails, please follow the instructions in SRILM's INSTALL file. For instance, if SRILM's Makefile does not identify that your're running a 64 bit Linux you might have to run "make MACHINE_TYPE=i686-m64 World".

After you successfully compile SRILM, Joshua will need to know what directory it is in. You can type pwd to get the absolute path to the sirlm/ directory that you created. Once you've figured out the path, set an SRILM environment variable by typing:

export SRILM="/path/to/srilm"

Where "/path/to/srilm" is replaced with your path. You'll also need to set a JAVA_HOME environment variable. For Mac OS X this usually is done by typing:

export JAVA_HOME="/Library/Java/Home"

These variables will need to be set every time you use Joshua, so it's useful to add them to your .bashrc, .bash_profile or .profile file.

Download and Install Joshua

First, download the Joshua release 1.3. tar file. Next, type the following commands to untar the file and compile the Java classes:

tar xfz joshua.tar.gz
cd joshua
ant

Running ant will compile the Java classes and link in srilm. If everything works properly, you should see the message BUILD SUCCESSFUL. If you get a BUILD FAILED message, it may be because you have not properly set the paths to SRILM and JAVA_HOME, or because srilm was not compiled properly, as described above.

For the examples in this document, you will need to set a JOSHUA environment variable:

export JOSHUA="/path/to/joshua/trunk"

Run the example model

To test to make sure that the decoder is installed properly, we'll translate 5 sentences using a small translation model that loads quickly. The sentences that we will translate are contained in example/example.test.in

The small translation grammar contains 15,939 rules -- you can get the count of the number of rules by running gunzip -c example/example.hiero.tm.gz | wc -l or you can see the first few translation rules with gunzip -c example/example.hiero.tm.gz | head

[X] ||| [X,1] 科学家 [X,2] ||| [X,1] scientists to [X,2] ||| 2.17609119 0.333095818 1.53173875
[X] ||| [X,1] 科学家 [X,2] ||| [X,2] of the [X,1] scientists ||| 2.47712135 0.333095818 2.17681264
[X] ||| [X,1] 科学家 [X,2] ||| [X,2] of [X,1] scientists ||| 2.47712135 0.333095818 1.13837981
[X] ||| [X,1] 科学家 [X,2] ||| [X,2] [X,1] scientists ||| 2.47712135 0.333095818 0.218843221
[X] ||| [X,1] 科学家 [X,2] ||| [X,1] scientists [X,2] ||| 1.01472330 0.333095818 0.218843221
[X] ||| [X,1] 科学家 [X,2] ||| [X,2] of scientists of [X,1] ||| 2.47712135 0.333095818 2.05791640
[X] ||| [X,1] 科学家 [X,2] ||| scientists [X,1] for [X,2] ||| 2.47712135 0.333095818 2.05956721
[X] ||| [X,1] 科学家 [X,2] ||| [X,1] scientist [X,2] ||| 1.63202321 0.303409695 0.977472364
[X] ||| [X,1] 科学家 [X,2] ||| [X,1] scientists , [X,2] ||| 2.47712135 0.333095818 1.68990576
[X] ||| [X,1] 科学家 [X,2] ||| scientists [X,2] [X,1] ||| 2.47712135 0.333095818 0.218843221

The different parts of the rules are separated by the ||| delimiter. The first part of the rule is the left-hand side non-terminal. The second and third parts are the right-hand side. The three numbers listed after each translation rules are negative log probabilities that signify, in order:

• prob(e|f) - the probability of the English phrase given the foreign phrase
• lexprob(e|f) - the lexical translation probabilities of the English words given the foreign words
• lexprob(f|e) - the lexical translation probabilities of the foreign words given the English words

You can use the grammar to translate the test set by running

java -Xmx1g -cp $JOSHUA/bin \
	-Djava.library.path=$JOSHUA/lib \
	-Dfile.encoding=utf8 joshua.decoder.JoshuaDecoder \
	example/example.config.srilm \
	example/example.test.in \
	example/example.nbest.srilm.out

For those of you who aren't very familiar with Java, the arguments are the following:

• -Xmx1g -- this tells Java to use 1 GB of memory.
• -cp $JOSHUA/bin -- this specifies the directory that contains the Java class files.
• -Djava.library.path=$JOSHUA/lib -- this specifies the directory that contains the libraries that link in C++ code
• -Dfile.ecoding=utf8 -- this tells java to use unicode as the default file encoding.
• joshua.decoder.JoshuaDecoder -- This is the class that is run. If you want to look at the the source code for this class, you can find it in src/joshua/decoder/JoshuaDecoder.java
• example/example.config.srilm -- This is the configuration file used by Joshua.
• example/example.test.in -- This is the input file containing the sentences to translate.
• example/example.nbest.srilm.out -- This is the output file that the n-best translations will be written to.

You can inspect the output file by typing head example/example.nbest.srilm.out

0 ||| scientists to vital early 失智症 the chromosome completed has ||| -127.759 -6.353 -11.577 -5.325 -3.909 ||| -135.267
0 ||| scientists for vital early 失智症 the chromosome completed has ||| -128.239 -6.419 -11.179 -5.390 -3.909 ||| -135.556
0 ||| scientists to related early 失智症 the chromosome completed has ||| -126.942 -6.450 -12.716 -5.764 -3.909 ||| -135.670
0 ||| scientists to vital early 失智症 the chromosomes completed has ||| -128.354 -6.353 -11.396 -5.305 -3.909 ||| -135.714
0 ||| scientists to death early 失智症 the chromosome completed has ||| -127.879 -6.575 -11.845 -5.287 -3.909 ||| -135.803
0 ||| scientists as vital early 失智症 the chromosome completed has ||| -128.537 -6.000 -11.384 -5.828 -3.909 ||| -135.820
0 ||| scientists for related early 失智症 the chromosome completed has ||| -127.422 -6.516 -12.319 -5.829 -3.909 ||| -135.959
0 ||| scientists for vital early 失智症 the chromosomes completed has ||| -128.834 -6.419 -10.998 -5.370 -3.909 ||| -136.003
0 ||| scientists to vital early 失智症 completed the chromosome has ||| -127.423 -7.364 -11.577 -5.325 -3.909 ||| -136.009
0 ||| scientists to vital early 失智症 of chromosomes completed has ||| -127.427 -7.136 -11.612 -5.816 -3.909 ||| -136.086

This file contains the n-best translations, under the model. The first 10 lines that you see above are 10 best translations of the first sentence. Each line contains 4 fields. The first field is the index of the sentence (index 0 for the first sentence), the second field is the translation, the third field contains the each of the individual feature function scores for the translation (language model, rule translation probability, lexical translation probability, reverse lexical translation probability, and word penalty), and the final field is the overall score.

To get the 1-best translations for each sentence in the test set without all of the extra information, you can run the following command:

java -Xmx1g -cp $JOSHUA/bin \
	-Dfile.encoding=utf8 joshua.util.ExtractTopCand \
	example/example.nbest.srilm.out \
	example/example.nbest.srilm.out.1best

You cat then look at the 1-best output file by typing cat example/example.nbest.srilm.out.1best:

scientists to vital early 失智症 the chromosome completed has
( , paris 2 ) international a group of scientists said that they completed to human to chromosome 14 has , the chromosome with many diseases , including more years , may with the early 阿耳滋海默氏症 .
this is to now completed has in the fourth chromosome , which 八千七百多万 to carry when ( dna ) .
the weekly british science the study showed that the chromosome 14 are by 一千零五十 genes and gene fragments .
the goal of gene scientists is to provide diagnostic tools to found of the flawed genes , are still provide a to stop these genes treatments .

If your translations are identical to the ones above then Joshua is installed correctly. With this small model, there are many untranslated words, and the quality of the translations is very low. In the next steps, we'll show you how to train a model for a new language pair, using a larger training corpus that will result in higher quality translations.

There are several sources for training data. A good source of free parallel corpora for European languages is the Europarl corpus that is distributed as part of the Workshop on Statistical Machine Translation. If you sign up to participate in the annual NIST Open Machine Translation Evaluation you can get access large Arabic-English and Chinese-English parallel corpora, and a small Urdu-English parallel corpus.

Once you've gathered your data, you will need to do several preprocess steps: sentence alignment, tokenization, normalization, and subsampling.

Sentence alignment

In this exercise, we'll start with an existing sentence-aligned parallel corpus. Download this tarball, which contains a Spanish-Engish parallel corpus, along with a dev and a test set: data.tar.gz

The data tarball contains two training directories training/, which includes a subset of the corpus, and full-training, which includes the full corpus. I strongly recommend staring with the smaller set, and building an end-to-end system with it, since many steps take a very long time on the full data set. You should debug on the smaller set to avoid wasting time.

If you start with your own data set, you will need to sentence align it. We recommend Bob Moore's bilingual sentence aligner.

Tokenization

Joshua uses whitespace to delineate words. For many languages, tokenization can be as simple as separating punctation off as its own token. For languages like Chinese, which don't put spaces around words, tokenization can be more tricky.

For this example we'll use the simple tokenizer that is released as part of the WMT. It's located in the tarball under the scripts directory. To use it type the following commands:

tar xfz data.tar.gz

cd data/

gunzip -c es-en/full-training/europarl-v4.es-en.es.gz \
	| perl scripts/tokenizer.perl -l es \
	> es-en/full-training/training.es.tok

gunzip -c es-en/full-training/europarl-v4.es-en.en.gz \
	| perl scripts/tokenizer.perl -l en \
	> es-en/full-training/training.en.tok