A Hybrid Approach to Skeleton-based Translation

The first issue that arises is how to identify the skeleton for a given source sentence. Many ways are available. E.g., we can start with a full syntactic tree and transform it into a simpler form (e.g., removing a sub-tree). Here we choose a simple and straightforward method: a skeleton is obtained by dropping all unimportant words in the original sentence, while preserving the grammaticality. See the following for an example skeleton of a Chinese sentence.

Original Sentence (subscripts represent indices):

ÿ${}_{\left[1\right]}$

per

¶Ö${}_{\left[2\right]}$

ton

º£Ë®µ­»¯${}_{\left[3\right]}$

seawater desalination

´¦Àí${}_{\left[4\right]}$

treatment

µÄ${}_{\left[5\right]}$

of

³É±¾${}_{\left[6\right]}$

the cost

ÔÚ${}_{\left[7\right]}$

5${}_{\left[8\right]}$

5

Ôª${}_{\left[9\right]}$

yuan

µÄ${}_{\left[10\right]}$

of

»ù´¡${}_{\left[11\right]}$

from

ÉÏ${}_{\left[12\right]}$

½øÒ»²½${}_{\left[13\right]}$

has been further

Ͻµ${}_{\left[14\right]}$

reduced

¡£${}_{\left[15\right]}$

.

(The cost of seawater desalination treatment has been further reduced from 5 yuan per ton.)

Sentence Skeleton (subscripts represent indices):

³É±¾${}_{\left[6\right]}$

the cost

½øÒ»²½${}_{\left[13\right]}$

has been further

Ͻµ${}_{\left[14\right]}$

reduced

¡£${}_{\left[15\right]}$

.

(The cost has been further reduced.)

Obviously the skeleton used in this work can be viewed as a simplified sentence. Thus the problem is in principle the same as sentence simplification/compression. The motivations of defining the problem in this way are two-fold. First, as the skeleton is a well-formed (but simple) sentence, all current MT approaches are applicable to the skeleton translation problem. Second, obtaining simplified sentences by word deletion is a well-studied issue  [Knight and Marcu2000, Clarke and Lapata2006, Galley and McKeown2007, Cohn and Lapata2008, Yamangil and Shieber2010, Yoshikawa et al.2012]. Many good sentence simpliciation/compression methods are available to our work. Due to the lack of space, we do not go deep into this problem. In Section 3.1 we describe the corpus and system employed for automatic generation of sentence skeletons.

Next we describe our approach to integrating skeleton information into MT models. We start with an assumption that the 1-best skeleton is provided by the skeleton identification system. Then we define skeleton-based translation as a task of searching for the best target string $\widehat{t}$ given the source string and its skeleton $\tau$:

As is standard in SMT, we further assume that 1) the translation process can be decomposed into a derivation of phrase-pairs (for phrase-based models) or translation rules (for syntax-based models); 2) and a linear function $g(\cdot )$ is used to assign a model score to each derivation. Let $d_{s,\tau ,t}$ (or $d$ for short) denote a translation derivation. The above problem can be redefined in a Viterbi fashion - we find the derivation $\widehat{d}$ with the highest model score given $s$ and $\tau$:

In this way, the MT output can be regarded as the target-string encoded in $\widehat{d}$.

To compute $g\left(d\right)$, we use a linear combination of a skeleton translation model $g_{skel}\left(d\right)$ and a full translation model $g_{full}\left(d\right)$:

where the skeleton translation model handles the translation of the sentence skeleton, while the full translation model is the baseline model and handles the original problem of translating the whole sentence. The motivation here is straightforward: we use an additional score $g_{skel}\left(d\right)$ to model the problem of skeleton translation and interpolate it with the baseline model. See Figure 1 for an example of applying the above model to phrase-based MT. In the figure, each source phrase is translated into a target phrase, which is represented by linked rectangles. The skeleton translation model focuses on the translation of the sentence skeleton, i.e., the solid (red) rectangles; while the full translation model computes the model score for all those phrase-pairs, i.e., all solid and dashed rectangles.

Another note on the model. Eq. (3) provides a very flexible way for model selection. While we will restrict ourself to phrase-based translation in the following description and experiments, we can choose different models/features for $g_{skel}\left(d\right)$ and $g_{full}\left(d\right)$. E.g., one may introduce syntactic features into $g_{skel}\left(d\right)$ due to their good ability in capturing structural information; and employ a standard phrase-based model for $g_{full}\left(d\right)$ in which not all segments of the sentence need to respect syntactic constraints.

In this work both the skeleton translation model $g_{skel}\left(d\right)$ and full translation model $g_{full}\left(d\right)$ resemble the usual forms used in phrase-based MT, i.e., the model score is computed by a linear combination of a group of phrase-based features and language models. In phrase-based MT, the translation problem is modeled by a derivation of phrase-pairs. Given a translation model $m$, a language model $lm$ and a vector of feature weights $𝐰$, the model score of a derivation $d$ is computed by

where ${𝐟}_m\left(d\right)$ is a vector of feature values defined on $d$, and ${𝐰}_m$ is the corresponding weight vector. $lm\left(d\right)$ and $w_{lm}$ are the score and weight of the language model, respectively.

To ease modeling, we only consider skeleton-consistent derivations in this work. A derivation $d$ is skeleton-consistent if no phrases in $d$ cross skeleton boundaries (e.g., a phrase where two of the source words are in the skeleton and one is outside). Obviously, from any skeleton-consistent derivation $d$ we can extract a skeleton derivation $d_{\tau }$ which covers the sentence skeleton exactly. For example, in Figure 1, the derivation of phrase-pairs $\left\{p_1,p_2,\mathrm{\dots },p_9\right\}$ is skeleton-consistent, and the skeleton derivation is formed by $\left\{p_1,p_4,p_5,p_9\right\}$.

Then, we can simply define $g_{skel}\left(d\right)$ and $g_{full}\left(d\right)$ as the model scores of $d_{\tau }$ and $d$:

This model makes the skeleton translation and full translation much simpler because they perform in the same way of string translation in phrase-based MT. Both $g_{skel}\left(d\right)$ and $g_{full}\left(d\right)$ share the same translation model $m$ which can easily learned from the bilingual data¹¹In $g_{skel}\left(d\right)$, we compute the reordering model score on the skeleton though it is learned from the full sentences. In this way the reordering problems in skeleton translation and full translation are distinguished and handled separately.. On the other hand, it has different feature weight vectors for individual models (i.e., $𝐰$ and ${𝐰}^{\tau }$).

For language modeling, $lm$ is the standard $n$-gram language model adopted in the baseline system. $l{m}^{\tau }$ is a skeletal language for estimating the well-formedness of the translation skeleton. Here a translation skeleton is a target string where all segments of non-skeleton translation are generalized to a symbol X. E.g., in Figure 1, the translation skeleton is ’the cost X has been further reduced X .’, where two Xs represent non-skeleton segments in the translation. In such a way of string representation, the skeletal language model can be implemented as a standard $n$-gram language model, that is, a string probability is calculated by a product of a sequence of $n$-gram probabilities (involving normal words and X). To learn the skeletal language model, we replace non-skeleton parts of the target sentences in the bilingual corpus to Xs using the source sentence skeletons and word alignments. The skeletal language model is then trained on these generalized strings in a standard way of $n$-gram language modeling.

By substituting Eq. (4) into Eqs. (5) and (6), and then Eqs. (3) and (2), we have the final model used in this work:

Figure 1 shows the translation process and associated model scores for the example sentence. Note that this method does not require any new translation models for implementation. Given a baseline phrase-based system, all we need is to learn the feature weights $𝐰$ and ${𝐰}^{\tau }$ on the development set (with source-language skeleton annotation) and the skeletal language model $l{m}^{\tau }$ on the target-language side of the bilingual corpus. To implement Eq. (7), we can perform standard decoding while ”doubly weighting” the phrases which cover a skeletal section of the sentence, and combining the two language models and the translation model in a linear fashion.

We experimented with our approach on Chinese-English translation using the NiuTrans open-source MT toolkit [Xiao et al.2012]. Our bilingual corpus consists of 2.7M sentence pairs. All these sentences were aligned in word level using the GIZA++ system and the ”grow-diag-final-and” heuristics. A 5-gram language model was trained on the Xinhua portion of the English Gigaword corpus in addition to the target-side of the bilingual data. This language model was used in both the baseline and our improved systems. For our skeletal language model, we trained a 5-gram language model on the target-side of the bilingual data by generalizing non-skeleton segments to Xs. We used the newswire portion of the NIST MT06 evaluation data as our development set, and used the evaluation data of MT04 and MT05 as our test sets. We chose the default feature set of the NiuTrans.Phrase engine for building the baseline, including phrase translation probabilities, lexical weights, a 5-gram language model, word and phrase bonuses, a ME-based lexicalized reordering model. All feature weights were learned using minimum error rate training [Och2003].

Our skeleton identification system was built using the t3 toolkit²²http://staffwww.dcs.shef.ac.uk/people/T.Cohn/t3/ which implements a state-of-the-art sentence simplification system. We used the NEU Chinese sentence simplification (NEUCSS) corpus as our training data [Zhang et al.2013]. It contains the annotation of sentence skeleton on the Chinese-language side of the Penn Parallel Chinese-English Treebank (LDC2003E07). We trained our system using the Parts 1-8 of the NEUCSS corpus and obtained a 65.2% relational F1 score and 63.1% compression rate in held-out test (Part 10). For comparison, we also manually annotated the MT development and test data with skeleton information according to the annotation standard provided within NEUCSS.

Table 1 shows the case-insensitive IBM-version BLEU and TER scores of different systems. We see, first of all, that the MT system benefits from our approach in most cases. In both the manual and automatic identification of sentence skeleton (rows 2 and 4), there is a significant improvement on the ”All” data set. However, using different skeleton identification results for training and inference (row 3) does not show big improvements due to the data inconsistency problem.

Another interesting question is whether the skeletal language model really contributes to the improvements. To investigate it, we removed the skeletal language model from our skeleton-based translation system (with automatic skeleton identification on both the development and test sets). Seen from row $-l{m}^{\tau }$ of Table 1, the removal of the skeletal language model results in a significant drop in both BLEU and TER performance. It indicates that this language model is very beneficial to our system. For comparison, we removed the skeleton-based translation model from our system as well. Row $-m^{\tau }$ of Table 1 shows that the skeleton-based translation model can contribute to the overall improvement but there is no big differences between baseline and $-m^{\tau }$.

Apart from showing the effects of the skeleton-based model, we also studied the behavior of the MT system under the different settings of search space. Row s-space of Table 1 shows the BLEU and TER results of restricting the baseline system to the space of skeleton-consistent derivations, i.e., we remove both the skeleton-based translation model and language model from the SBMT system. We see that the limited search space is a little harmful to the baseline system. Further, we regarded skeleton-consistent derivations as an indicator feature and introduced it into the baseline system. Seen from row s-feat., this feature does not show promising improvements. These results indicate that the real improvements are due to the skeleton-based model/features used in this work, rather than the ”well-formed” derivations.