The Kyoto Free Translation Task (KFTT)

by Graham Neubig (my last name at gmail.com)

日本語

Welcome to the home of the Kyoto Free Translation Task (KFTT). The KFTT is a task for the evaluation and development of Japanese-English machine translation systems. It is designed to allow for free, simple to start, reproducible, and progressive research. Details about the task's principles, the features of the data being used, how to start, and more are listed below.

Principles

The Kyoto Free Translation Task was designed around four principles. That research should be free, should be simple, should be reproducible, and should be progressive. Specifically:

Data Details

Original Data

The Kyoto Free Translation Task is a task for Japanese-English translation that focuses on Wikipedia articles related to Kyoto. The data used was originally prepared by the National Institute for Information and Communication Technology (NICT) and released as the Japanese-English Bilingual Corpus of Wikipedia's Kyoto Articles (we are simply using the data, NICT does not specifically endorse or sponsor this task). The features of the data are as follows:

Data Processing

In order to make it more simple to perform machine translation on this data, performed a number of processing steps.

  1. Extracting the data into a format usable by most machine translation systems.
  2. Tokenizing the data into words. English tokenizing was performed using scripts included with the Moses toolkit, and Japanese tokenization was performed using a model for KyTea that was lightly adapted to the Kyoto domain (the model is included in the package).
  3. Separating the data into training, tuning, development, and test sets. The training data should be used for training statistical models, tuning data used for tuning weights, development data used for testing the system in development, and testing data used for reporting final results.
  4. Cleaning the training data to remove sentences with less than 1 or more than 40 words.

After this processing, the size of the data is as follows (ver. 1.0):

ArticlesSentencesJapanese WordsEnglish Words
Train 14126440k12.0M11.5M
Train (clean)14126330k6.09M5.91M
Tune 15 123534.4k30.8k
Dev 15 116626.8k24.3k
Test 15 116028.5k26.7k

Getting Started

Data + Baseline System

In order to get started on the Kyoto Free Translation Task, you need to do three things, download the data, install the third party software that makes it work, and run the script that compiles the data and trains a baseline system. The data and compilation scripts can be downloaded here:

The two current baselines are the KFTT Moses Baseline, v. 1.4, which is a standard Moses setup, and KFTT lader, v. 1.0 which uses lader for pre-ordering. Using lader will give you the best results, but is somewhat computationally intensive.

Previous versions: v. 1.3 v. 1.2 v. 1.1 v. 1.0

Please note that this data is distributed under the Creative Commons Attribution-Share-Alike License 3.0 license. Next, you need to download and install the third party software that makes the system work.

You should also set two environmental variables:

Once you have finished installing the third party software, expand the KFTT's .tar.gz file and run process.sh. Note that if you have installed any of the third party tools in an unusual place, you can modify process.sh to point to them appropriately. Also, the training may take a significant amount of time (24 hours?), so try to run it on a fast machine with several GB of memory if possible.

$ tar -xzf kfft-XXX.tar.gz
$ cd kfft-XXX
$ nohup ./process.sh &> process.log &

When the script has finished running, if everything has gone well, you will see a report of the final BLEU scores at the end of process.log, which should match the reported scores. If you get stuck please feel free to contact me (Graham) at any time.

Data Only

When using the Kyoto Free Translation Task for research, we recommend that you go through the steps to build the full baseline system. But if you only need the data for whatever reason, it can be downloaded via the below link.

Kyoto Free Translation Task (Data Only v. 1.0)

Segmenter Only

If you would just like the word segmenter used for Japanese tokenization, download KyTea (v. 0.4.0+) and use it with the KFTT segmentation model.

Alignment Data

If you would like to test the accuracy of a word alignment method on the KFTT data, or would like to train a supervised alignment system, you can use the following data. All alignments were created by two annotators then checked for consistency over 1235 sentences of the tuning set.

Kyoto Free Translation Task Japanese-English Alignment Data

Tracks (Leader Board)

The goal of this task is to create a standard way for people to compare and improve Japanese-English translation systems. In order to do so, we are taking a friendly competition format, where participants try to compete to improve results. There are two tracks:

Results are measured by BLEU Score. Starting at version 1.3, both English and Japanese results are measured with lowercased, tokenized text (previously English results were detokenized and cased text, and are shown in light grey).

Open

Japanese -> English
DateSystem NameParticipantsInstitutiondev BLEUtest BLEUComment
2012-9-2KyTea/GIZA++/Moses/Lader 1.0Graham NeubigNAIST16.9319.35KyTea/GIZA++/Moses with preordering using lader
2012-4-9KyTea/GIZA++/Moses 1.3/1.4Graham NeubigNAIST15.4117.68KyTea v. 0.4.1/GIZA++/Moses baseline
2012-2-4KyTea/GIZA++/Moses 1.2Graham NeubigKyoto University9.4010.53KyTea v. 0.4.0/GIZA++/Moses baseline
2011-5-16KyTea/GIZA++/Moses 1.1Graham NeubigKyoto University8.9810.58KyTea v. 0.3.0/GIZA++/Moses baseline

English -> Japanese
DateSystem NameParticipant NamesInstitutiondev BLEUtest BLEUComment
2012-9-2KyTea/GIZA++/Moses/Lader 1.0Graham NeubigNAIST21.0823.15KyTea/GIZA++/Moses with preordering using lader
2012-4-9KyTea/GIZA++/Moses v. 1.3/1.4Graham NeubigNAIST19.2421.03KyTea v. 0.4.1/GIZA++/Moses baseline
2012-2-4KyTea/GIZA++/Moses v. 1.2Graham NeubigKyoto University19.0020.85KyTea v. 0.4.0/GIZA++/Moses baseline
2011-5-16KyTea/GIZA++/Moses v. 1.1Graham NeubigKyoto University18.7020.32KyTea v. 0.3.0/GIZA++/Moses baseline

Closed

Japanese -> English
DateSystem NameParticipantsInstitutiondev BLEUtest BLEUComment
2011-2-18Google TranslateGraham NeubigKyoto University5.255.27Google Translate results from 2011-2-18.
2011-2-18Excite TranslateGraham NeubigKyoto University3.834.31Excite Translate results from 2011-2-18.

English -> Japanese
DateSystem NameParticipant NamesInstitutiondev BLEUtest BLEUComment
2011-2-18Google TranslateGraham NeubigKyoto University11.4311.53Google Translate results from 2011-2-18. Japanese results were re-segmented using KyTea for evaluation.
2011-2-18Excite TranslateGraham NeubigKyoto University6.407.25Excite Translate results from 2011-2-18. Japanese results were re-segmented using KyTea for evaluation.

References/Citations

This section contains references for this task and the systems performing in it.

Task

If you would like to cite this task when writing a paper, please use the following information:

@misc{neubig11kftt,
	author = {Graham Neubig},
	title = {The {Kyoto} Free Translation Task},
	howpublished = {http://www.phontron.com/kftt},
	year = {2011}
}

Systems

If you perform research using this task, please contact me and I will list it here.

Version History

KyTea/GIZA++/Moses Version 1.4 (2013-5-11)

In version 1.3 some zero-length sentences were extracted from the XML file, so now only sentences with length greater than zero are extracted (thanks to Tetsuo Kiso for the patch!).

KyTea/GIZA++/Moses/Lader Version 1.0 (2012-9-2)

This version introduces the pre-ordering of Neubig et al. (2012) as implemented by lader, which significantly improves over the Moses baseline.

KyTea/GIZA++/Moses Version 1.3 (2012-4-9)

Previous versions evaluated English results using recased and detokenized text, but this led to unstable and extremely low BLEU scores. As a result versions 1.3 and up will use tokenized and uncased text.

KyTea/GIZA++/Moses Version 1.2 (2012-2-4)

Upgraded the task to work with KyTea version 0.4.0.

KyTea/GIZA++/Moses Version 1.1 (2011-5-16)

Version 1.0 didn't work with the latest version of KyTea, so this is fixed (thanks to Atsushi Fujita for pointing this out).

Version 1.0 (2011-2-18)

The initial release of the task.