Learning Topic Representation for SMT with Neural Networks^††thanks: This work was done while the first and fourth authors were visiting Microsoft Research.

In the pre-training phase, we leverage neural network structures to transform high-dimensional sparse vectors to low-dimensional dense vectors. The topic similarity is calculated on top of the learned dense vectors. This dense representation should preserve the information from the bag-of-words input, meanwhile alleviate data sparse problem. Therefore, we use a specially designed mechanism called auto-encoder to solve this problem. Auto-encoder [1] is one of the basic building blocks of deep learning. Assuming that the input is a $n$-of-$V$ binary vector x representing the bag-of-words ($V$ is the vocabulary size), an auto-encoder consists of an encoding process $g\left(\text{𝐱}\right)$ and a decoding process $h\left(g\left(\text{𝐱}\right)\right)$. The objective of the auto-encoder is to minimize the reconstruction error $ℒ\left(h\left(g\left(\text{𝐱}\right)\right),\text{𝐱}\right)$. Our goal is to learn a low-dimensional vector which can preserve information from the original $n$-of-$V$ vector.

One problem with auto-encoder is that it treats all words in the same way, making no distinguishment between function words and content words. The representation learned by auto-encoders tends to be influenced by the function words, thereby it is not robust. To alleviate this problem, Vincent et al. (2008) proposed the Denoising Auto-Encoder (DAE), which aims to reconstruct a clean, “repaired” input from a corrupted, partially destroyed vector. This is done by corrupting the initial input x to get a partially destroyed version $\widetilde{\text{𝐱}}$. DAE is capable of capturing the global structure of the input while ignoring the noise. In our task, for each sentence, we treat the retrieved $N$ relevant documents as a single large document and convert it to a bag-of-words vector x in Figure 2. With DAE, the input x is manually corrupted by applying masking noise (randomly mask 1 to 0) and getting $\widetilde{\text{𝐱}}$. Denoising training is considered as “filling in the blanks” [33], which means the masking components can be recovered from the non-corrupted components. For example, in IT related texts, if the word driver is masked, it should be predicted through hidden units in neural networks by active signals such as “buffer”, “user response”, etc.

In our case, the encoding process transforms the corrupted input $\widetilde{\text{𝐱}}$ into $g\left(\widetilde{\text{𝐱}}\right)$ with two layers: a linear layer connected with a non-linear layer. Assuming that the dimension of the $g\left(\widetilde{\text{𝐱}}\right)$ is $L$, the linear layer forms a $L\times V$ matrix $W$ which projects the $n$-of-$V$ vector to a $L$-dimensional hidden layer. After the bag-of-words input has been transformed, they are fed into a subsequent layer to model the highly non-linear relations among words:

where z is the output of the non-linear layer, b is a $L$-length bias vector. $f(\cdot )$ is a non-linear function, where common choices include sigmoid function, hyperbolic function, “hard” hyperbolic function, rectifier function, etc. In this work, we use the rectifier function as our non-linear function due to its efficiency and better performance [13]:

The decoding process consists of a linear layer and a non-linear layer with similar network structures, but different parameters. It transforms the $L$-dimensional vector $g\left(\widetilde{\text{𝐱}}\right)$ to a $V$-dimensional vector $h\left(g\left(\widetilde{\text{𝐱}}\right)\right)$. To minimize reconstruction error with respect to $\widetilde{\text{𝐱}}$, we define the loss function as the L2-norm of the difference between the uncorrupted input and reconstructed input:

Multi-layer neural networks are trained with the standard back-propagation algorithm [26]. The gradient of the loss function is calculated and back-propagated to the previous layer to update its parameters. Training neural networks involves many factors such as the learning rate and the length of hidden layers. We will discuss the optimization of these parameters in Section 4.

In the fine-tuning phase, we stack another layer on top of the two low-dimensional vectors to maximize the similarity between source and target languages. The similarity scores are integrated into the standard log-linear model for making translation decisions. Since the vectors from DAE are trained using information from monolingual training data independently, these vectors may be inadequate to measure bilingual topic similarity due to their different topic spaces. Therefore, in this stage, parallel sentence pairs are used to help connecting the vectors from different languages because they express the same topic. In fact, the objective of fine-tuning is to discover a latent topic space which is shared by both languages as much as possible. This shared topic space is particularly useful when the SMT decoder tries to match the source texts and translation candidates in the target language.

Given a parallel sentence pair $⟨f,e⟩$, the DAE learns representations for $f$ and $e$ respectively, as ${\text{𝐳}}_f=g\left(\text{𝐟}\right)$ and ${\text{𝐳}}_e=g\left(\text{𝐞}\right)$ in Figure 1. We then take two vectors as the input to calculate their similarity. Consequently, the whole neural network can be fine-tuned towards the supervised criteria with the help of parallel data. The similarity score of the representation pair $⟨{\text{𝐳}}_f,{\text{𝐳}}_e⟩$ is defined as the cosine similarity of the two vectors:

Since a parallel sentence pair should have the same topic, our goal is to maximize the similarity score between the source sentence and target sentence. Inspired by the contrastive estimation method [27], for each parallel sentence pair $⟨f,e⟩$ as a positive instance, we select another sentence pair $⟨f^{\prime },e^{\prime }⟩$ from the training data and treat $⟨f,e^{\prime }⟩$ as a negative instance. To make the similarity of the positive instance larger than the negative instance by some margin $\eta$, we utilize the following pairwise ranking loss:

where $\eta =\frac{1}{2}-sim\left(f,f^{\prime }\right)$. The rationale behind this criterion is, the smaller $sim\left(f,f^{\prime }\right)$ is, the more we should penalize negative instances.

To effectively train the model in this task, negative instances must be selected carefully. Since different sentences may have very similar topic distributions, we select negative instances that are dissimilar with the positive instances based on the following criteria:

1. 1.

   For each positive instance $⟨f,e⟩$, we select $e^{\prime }$ which contains at least 30% different content words from $e$.

2. 2.

   If we cannot find such $e^{\prime }$, remove $⟨f,e⟩$ from the training instances for network learning.

The model minimizes the pairwise ranking loss across all training instances:

We used standard back-propagation algorithm to further fine-tune the neural network parameters $W$ and b in Equation (1). The learned neural networks are used to obtain sentence topic representations, which will be further leveraged to infer topic representations of bilingual translation rules.

We incorporate the learned topic similarity scores into the standard log-linear framework for SMT. When a synchronous rule $⟨\alpha ,\gamma ⟩$ is extracted from a sentence pair $⟨f,e⟩$, a triple instance $ℐ=\left(⟨\alpha ,\gamma ⟩,⟨f,e⟩,c\right)$ is collected for inferring the topic representation of $⟨\alpha ,\gamma ⟩$, where $c$ is the count of rule occurrence. Following [7], we give a count of one for each phrase pair occurrence and a fractional count for each hierarchical phrase pair. The topic representation of $⟨\alpha ,\gamma ⟩$ is then calculated as the weighted average:

where $𝒯$ denotes all instances for the rule $⟨\alpha ,\gamma ⟩$, ${\text{𝐳}}_{\alpha }$ and ${\text{𝐳}}_{\gamma }$ are the source-side and target-side topic vectors respectively.

By measuring the similarity between the source texts and bilingual translation rules, the SMT decoder is able to encourage topic relevant translation candidates and penalize topic irrelevant candidates. Therefore, it helps to train a smarter translation model with the embedded topic information. Given a source sentence $s$ to be translated, we define the similarity as follows:

where ${\text{𝐳}}_s$ is the topic representation of $s$. The similarity calculated against ${\text{𝐳}}_{\alpha }$ or ${\text{𝐳}}_{\gamma }$ denotes the source-to-source or the source-to-target similarity.

We also consider the topic sensitivity estimation since general rules have flatter distributions while topic-specific rules have sharper distributions. A standard entropy metric is used to measure the sensitivity of the source-side of $⟨\alpha ,\gamma ⟩$ as:

where $z_{\alpha i}$ is a component in the vector ${\text{𝐳}}_{\alpha }$. The target-side sensitivity $Sen\left(\gamma \right)$ can be calculated in a similar way. The larger the sensitivity is, the more topic-specific the rule manifests.

In addition to traditional SMT features, we add new topic-related features into the standard log-linear framework. For the SMT system, the best translation candidate $\widehat{e}$ is given by:

where the translation probability is given by:

where ${\varphi }_j\left(f,e\right)$ is the standard feature function and $w_j$ is the corresponding feature weight. ${\varphi }_k\left(f,e\right)$ is the topic-related feature function and $w_k$ is the feature weight. The detailed feature description is as follows:

Standard features: Translation model, including translation probabilities and lexical weights for both directions (4 features), 5-gram language model (1 feature), word count (1 feature), phrase count (1 feature), NULL penalty (1 feature), number of hierarchical rules used (1 feature).

Topic-related features: rule similarity scores (2 features), rule sensitivity scores (2 features).

We evaluate the performance of our neural network based topic similarity model on a Chinese-to-English machine translation task. In neural network training, a large number of monolingual documents are collected in both source and target languages. The documents are mainly from two domains: news and weblog. We use Chinese and English Gigaword corpus (Version 5) which are mainly from news domain. In addition, we also collect weblog documents with a variety of topics from the web. The total data statistics are presented in Table 1. These documents are built in the format of inverted index using Lucene²²http://lucene.apache.org/, which can be efficiently retrieved by the parallel sentence pairs. The most relevant $N$ documents are collected, where we experiment with $N=\left\{1,5,10,20,50\right\}$.

We implement a distributed framework to speed up the training process of neural networks. The network is learned with mini-batch asynchronous gradient descent with the adaptive learning rate procedure called AdaGrad [11]. We use 32 model replicas in each iteration during the training. The model parameters are averaged after each iteration and sent to each replica for the next iteration. The vocabulary size for the input layer is 100,000, and we choose different lengths for the hidden layer as $L=\left\{100,300,600,1000\right\}$ in the experiments. In the pre-training phase, all parallel data is fed into two neural networks respectively for DAE training, where network parameters $W$ and b are randomly initialized. In the fine-tuning phase, for each parallel sentence pair, we randomly select other ten sentence pairs which satisfy the criterion as negative instances. These training instances are leveraged to optimize the similarity of two vectors.

In SMT training, an in-house hierarchical phrase-based SMT decoder is implemented for our experiments. The CKY decoding algorithm is used and cube pruning is performed with the same default parameter settings as in Chiang (2007). The parallel data we use is released by LDC³³LDC2003E14, LDC2002E18, LDC2003E07, LDC2005T06, LDC2005T10, LDC2005E83, LDC2006E34, LDC2006E85, LDC2006E92, LDC2006E26, LDC2007T09. In total, the datasets contain nearly 1.1 million sentence pairs. Translation models are trained over the parallel data that is automatically word-aligned using GIZA++ in both directions, and the diag-grow-final heuristic is used to refine symmetric word alignment. An in-house language modeling toolkit is used to train the 5-gram language model with modified Kneser-Ney smoothing [17]. The English monolingual data used for language modeling is the same as in Table 1. The NIST 2003 dataset is the development data. The testing data consists of NIST 2004, 2005, 2006 and 2008 datasets. The evaluation metric for the overall translation quality is case-insensitive BLEU4 [25]. The reported BLEU scores are averaged over 5 times of running MERT [24]. A statistical significance test is performed using the bootstrap re-sampling method [18].

The baseline is a re-implementation of the Hiero system [7]. The phrase pairs that appear only once in the parallel data are discarded because most of them are noisy. We also use the fix-discount method in Foster et al. (2006) for phrase table smoothing. This implementation makes the system perform much better and the translation model size is much smaller.

We compare our method with the LDA-based approach proposed by Xiao et al. (2012). In [34], the topic of each sentence pair is exactly the same as the document it belongs to. Since some of our parallel data does not have document-level information, we rely on the IR method to retrieve the most relevant document and simulate this approach. The PLDA toolkit [22] is used to infer topic distributions, which takes 34.5 hours to finish.

We illustrate the relationship among translation accuracy (BLEU), the number of retrieved documents ($N$) and the length of hidden layers ($L$) on different testing datasets. The results are shown in Figure 3. The best translation accuracy is achieved when $N$=10 for most settings. This confirms that enriching the source text with topic-related documents is very useful in determining topic representations, thereby help to guide the synchronous rule selection. However, we find that as $N$ becomes larger in the experiments, e.g. $N$=50, the translation accuracy drops drastically. As more documents are retrieved, less relevant information is also used to train the neural networks. Irrelevant documents bring so many unrelated topic words hence degrade neural network learning performance.

Another important factor is the length of hidden layers $L$ in the network. In deep learning, this parameter is often empirically tuned with human efforts. As shown in Figure 3, the translation accuracy is better when $L$ is relatively small. Actually, there is no obvious distinction of the performance when $L$ is less than 600. However, when $L$ equals 1,000, the translation accuracy is inferior to other settings. The main reason is that parameters in the neural networks are too many to be effectively trained. As we know when $L$=1000, there are a total of $100,000\times 1,000$ parameters between the linear and non-linear layers in the network. Limited training data prevents the model from getting close to the global optimum. Therefore, the model is likely to fall in local optima and lead to unacceptable representations.

We evaluate the performance of adding new topic-related features to the log-linear model and compare the translation accuracy with the method in [34]. To make different methods comparable, we set the dimension of topic representation as 100 for all settings. This takes 10 hours in pre-training phase and 22 hours in fine-tuning phase. Table 2 shows how the accuracy is improved with more features added. The results confirm that topic information is indispensable for SMT since both [34] and our neural network based method significantly outperforms the baseline system. Our method improves 0.86 BLEU points at most and 0.76 BLEU points on average over the baseline. We observe that source-side similarity is more effective than target-side similarity, but their contributions are cumulative. This proves that bilingually induced topic representation with neural network helps the SMT system disambiguate translation candidates. Furthermore, rule sensitivity features improve SMT performance compared with only using similarity features. Because topic-specific rules usually have a larger sensitivity score, they can beat general rules when they obtain the same similarity score against the input sentence. Finally, when all new features are integrated, the performance is the best, preforming substantially better than [34] with 0.39 BLEU points on average.

It is worth mentioning that the performance of [34] is similar to the settings with $N$=1 and $L$=100 in Figure 3. This is not simply coincidence since we can interpret their approach as a special case in our neural network method: when a parallel sentence pair has document-level information, that document will be retrieved for training; otherwise, the most relevant document will be retrieved from the monolingual data. Therefore, our method can be viewed as a more general framework than previous LDA-based approaches.

In this section, we give a case study to explain why our method works. An example of translation rule disambiguation for a sentence from the NIST 2005 dataset is shown in Figure 4. We find that the topic of this sentence is about “rescue after a natural disaster”. Under this topic, the Chinese rule “发送 X” should be translated to “deliver X” or “distribute X”. However, the baseline system prefers “send X” rather than those two candidates. Although the translation probability of “send X” is much higher, it is inappropriate in this context since it is usually used in IT texts. For example, $⟨$发送邮件, send emails$⟩$, $⟨$发送信息, send messages$⟩$ and $⟨$发送数据, send data$⟩$. In contrast, with our neural network based approach, the learned topic distributions of “deliver X” or “distribute X” are more similar with the input sentence than “send X”, which is shown in Figure 4. The similarity scores indicate that “deliver X” and “distribute X” are more appropriate to translate the sentence. Therefore, adding topic-related features is able to keep the topic consistency and substantially improve the translation accuracy.