Improving machine translation by training against an automatic semantic frame based evaluation metric

Chi-kiu LO, Karteek Addanki, Markus Saers and Dekai Wu

The 51st Annual Meeting of the Association for Computational Linguistics - Short Papers (ACL Short Papers 2013)
Sofia, Bulgaria, August 4-9, 2013

---

Abstract

We present the first ever results showing that tuning a machine translation system against a semantic frame based objective function, MEANT, produces more robustly adequate translation than tuning against BLEU or TER as measured across commonly used metrics and human subjective evaluation. Moreover, even for informal web forum data, human evaluators preferred MEANT-tuned systems over BLEU- or TER-tuned systems by a significantly wider margin than for formal newswire---even though automatic semantic parsing might be expected to fare worse on informal language. We argue that by preserving the meaning of the translations as captured by semantic frames right in the training process, an MT system is constrained to make more accurate choices of both lexical and reordering rules. As a result, MT systems tuned against semantic frame based MT evaluation metrics produce output that is more adequate. Our approach is independent of the translation model paradigm, so, any translation model can benefit from the semantic knowledge incorporated through our approach. In this paper, we show for the first time that tuning an MT system against MEANT significantly improves translation adequacy on formal, as well as informal compared to tuning against BLEU or TER.

---

START Conference Manager (V2.61.0 - Rev. 2792M)