Learning New Semi-Supervised Deep Auto-encoder Features
for Statistical Machine Translation

We assume that source phrase $f=f_1,\mathrm{\cdots },f_{l_f}$ and target phrase $e=e_1,\mathrm{\cdots },e_{l_e}$ include $l_f$ and $l_e$ words, respectively. Following [Maskey and Zhou2012], we use the following 4 phrase features of each phrase pair [Koehn et al.2003] in the phrase table as the first type of input features, bidirectional phrase translation probability ($P\left(e\right|f)$ and $P\left(f\right|e)$), bidirectional lexical weighting ($Lex\left(e\right|f)$ and $Lex\left(f\right|e)$),

Zhao et al. (2004) proposed a way of using term weight based models in a vector space as additional evidences for phrase pair translation quality. This model employ phrase pair similarity to encode the weights of content and non-content words in phrase translation pairs. Following [Zhao et al.2004], we calculate bidirectional phrase pair similarity using cosine distance and BM25 distance as,

where, $p\left(e_j\right|f_i)$ and $p\left(f_i\right|e_j)$ represents bidirectional word translation probability. $w_{f_i}$ and $w_{e_j}$ are term weights for source and target words, $w_a^{e_j}$ and $w_a^{f_i}$ are the transformed weights mapped from all source/target words to the target/source dimension at word $e_j$ and $f_i$, respectively.

where, $k_1$, $b$, $k_3$ are set to be 1, 1 and 1000, respectively. $K=k_1\left(\left(1-b\right)+J/avg\left(l\right)\right)$, and $J$ is the phrase length ($l_e$ or $l_f$), $avg\left(l\right)$ is the average phrase length. Thus, we have the second type of input features

We adapt and extend bidirectional phrase generative probabilities as the input features, which have been used for domain adaptation [Foster et al.2010]. According to the background LMs, we estimate the bidirectional (source/target side) forward and backward phrase generative probabilities as

where, the bidirectional forward and backward¹¹Backward LM has been introduced by Xiong et al. (2011), which successfully capture both the preceding and succeeding contexts of the current word, and we estimate the backward LM by inverting the order in each sentence in the training data from the original order to the reverse order. background 4-gram LMs are trained by the corresponding side of bilingual corpus²²This corpus is used to train the translation model in our experiments, and we will describe it in detail in section 5.1.. Then, we have the third type of input features

We consider bidirectional phrase frequency as the input features, and estimate them as

where, the $count\left(f\right)$/$count\left(e\right)$ are the total number of phrase f/e appearing in the source/target side of the bilingual corpus, and the denominator are the total number of the phrases whose length are equal to $\left|f\right|$/$\left|e\right|$, respectively. Then, we have the forth type of input features

Phrase length plays an important role in the translation process [Koehn2010, Hopkins and May2011]. We normalize bidirectional phrase length by the maximum phrase length, and introduce them as the last type of input features

In summary, except for the first type of phrase feature $X_1$ which is used by [Maskey and Zhou2012], we introduce another four types of effective phrase features $X_2$, $X_3$, $X_4$ and $X_5$. Now, the input original phrase features $X$ includes 16 features in our experiments, as follows,

We build the DAE network where the first layer with visible nodes equaling to 16, and each visible node $v_i$ corresponds to the above original features $X$ in each phrase pair.

Inspired by [Maskey and Zhou2012], we first learn a deep generative model for feature learning using DBN. DBN is composed of multiple layers of latent variables with the first layer representing the visible feature vectors, which is built with stacked sets of RBMs [Hinton2002].

For a RBM, there is full connectivity between layers, but no connections within either layer. The connection weight $W$, hidden layer biases $c$ and visible layer biases $b$ can be learned efficiently using the contrastive divergence [Hinton2002, Carreira-Perpinan and Hinton2005]. When given a hidden layer $h$, factorial conditional distribution of visible layer $v$ can be estimated by

where $\sigma$ denotes the logistic sigmoid. Given $v$, the element-wise conditional distribution of $h$ is

The two conditional distributions can be shown to correspond to the generative model,

where,

After learning the first RBM, we treat the activation probabilities of its hidden units, when they are being driven by data, as the data for training a second RBM. Similarly, a $n_{th}$ RBM is built on the output of the $n-1_{th}$ one and so on until a sufficiently deep architecture is created. These $n$ RBMs can then be composed to form a DBN in which it is easy to infer the states of the $n_{th}$ layer of hidden units from the input in a single forward pass [Hinton et al.2006], as shown in Figure 1. This greedy, layer-by-layer pre-training can be repeated several times to learn a deep, hierarchical model (DBN) in which each layer of features captures strong high-order correlations between the activities of features in the layer below.

To deal with real-valued input features $X$ in our task, we use an RBM with Gaussian visible units (GRBM) [Dahl et al.2012] with a variance of 1 on each dimension. Hence, $P\left(v\right|h)$ and $E\left(v,h\right)$ in the first RBM of DBN need to be modified as

where $I$ is the appropriate identity matrix.

To speed-up the pre-training, we subdivide the entire phrase pairs (with features $X$) in the phrase table into small mini-batches, each containing 100 cases, and update the weights after each mini-batch. Each layer is greedily pre-trained for 50 epochs through the entire phrase pairs. The weights are updated using a learning rate of 0.1, momentum of 0.9, and a weight decay of 0.0002 $\times$ weight $\times$ learning rate. The weight matrix $W$ are initialized with small random values sampled from a zero-mean normal distribution with variance 0.01.

After the pre-training, for each phrase pair in the phrase table, we generate the DBN features [Maskey and Zhou2012] by passing the original phrase features $X$ through the DBN using forward computation.

To learn a semi-supervised DAE, we first “unroll” the above n layer DBN by using its weight matrices to create a deep, 2n-1 layer network whose lower layers use the matrices to “encode” the input and whose upper layers use the matrices in reverse order to “decode” the input [Hinton and Salakhutdinov2006, Salakhutdinov and Hinton2009, Deng et al.2010], as shown in Figure 2. The layer-wise learning of DBN as above must be treated as a pre-training stage that finds a good region of the parameter space, which is used to initialize our DAE’s parameters. Starting in this region, the DAE is then fine-tuned using average squared error (between the output and input) back-propagation to minimize reconstruction error, as to make its output as equal as possible to its input.

For the fine-tuning of DAE, we use the method of conjugate gradients on larger mini-batches of 1000 cases, with three line searches performed for each mini-batch in each epoch. To determine an adequate number of epochs and to avoid overfitting, we fine-tune on a fraction phrase table and test performance on the remaining validation phrase table, and then repeat fine-tuning on the entire phrase table for 100 epochs.

We experiment with various values for the noise variance and the threshold, as well as the learning rate, momentum, and weight-decay parameters used in the pre-training, the batch size and epochs in the fine-tuning. Our results are fairly robust to variations in these parameters. The precise weights found by the pre-training do not matter as long as it finds a good region of the parameter space from which to start the fine-tuning.

The fine-tuning makes the feature representation in the central layer of the DAE work much better [Salakhutdinov and Hinton2009]. After the fine-tuning, for each phrase pair in the phrase table, we estimate our DAE features by passing the original phrase features $X$ through the “encoder” part of the DAE using forward computation.

To combine these learned features (DBN and DAE feature) into the log-linear model, we need to eliminate the impact of the non-linear learning mechanism. Following [Maskey and Zhou2012], these learned features are normalized by the average of each dimensional respective feature set. Then, we append these features for each phrase pair to the phrase table as extra features.

Although DAE can learn more powerful and abstract feature representation, the learned features usually have smaller dimensionality compared with the dimensionality of the input features, such as the successful use for handwritten digits recognition [Hinton and Salakhutdinov2006, Hinton et al.2006], information retrieval [Salakhutdinov and Hinton2009, Mirowski et al.2010], and speech spectrograms [Deng et al.2010]. Moreover, although we have introduced another four types of phrase features ($X_2$, $X_3$, $X_4$ and $X_5$), the only 16 features in $X$ are a bottleneck for learning large hidden layers feature representation, because it has limited information, the performance of the high-dimensional DAE features which are directly learned from single DAE is not very satisfactory.

To learn high-dimensional feature representation and to further improve the performance, we introduce a natural horizontal composition for DAEs that can be used to create large hidden layer representations simply by horizontally combining two (or more) DAEs [Baldi2012], as shown in Figure 3. Two single DAEs with architectures $16/m_1/16$ and $16/m_2/16$ can be trained and the hidden layers can be combined to yield an expanded hidden feature representation of size $m_1+m_2$, which can then be fed to the subsequent layers of the overall architecture. Thus, these new $m_1+m_2$-dimensional DAE features are added as extra features to the phrase table.

Differences in $m_1$- and $m_2$-dimensional hidden representations could be introduced by many different mechanisms (e.g., learning algorithms, initializations, training samples, learning rates, or distortion measures) [Baldi2012]. In our task, we introduce differences by using different initializations and different fractions of the phrase table.

We now test our DAE features on the following two Chinese-English translation tasks.

IWSLT. The bilingual corpus is the Chinese-English part of Basic Traveling Expression corpus (BTEC) and China-Japan-Korea (CJK) corpus (0.38M sentence pairs with 3.5/3.8M Chinese/English words). The LM corpus is the English side of the parallel data (BTEC, CJK and CWMT08³³the 4th China Workshop on Machine Translation) (1.34M sentences). Our development set is IWSLT 2005 test set (506 sentences), and our test set is IWSLT 2007 test set (489 sentences).

NIST. The bilingual corpus is LDC⁴⁴LDC2002E18, LDC2002T01, LDC2003E07, LDC2003E14, LDC2003T17, LDC2004T07, LDC2004T08, LDC2005T06, LDC2005T10, LDC2005T34, LDC2006T04, LDC2007T09 (3.4M sentence pairs with 64/70M Chinese/English words). The LM corpus is the English side of the parallel data as well as the English Gigaword corpus (LDC2007T07) (11.3M sentences). Our development set is NIST 2005 MT evaluation set (1084 sentences), and our test set is NIST 2006 MT evaluation set (1664 sentences).

We choose the Moses [Koehn et al.2007] framework to implement our phrase-based machine system. The 4-gram LMs are estimated by the SRILM toolkit with modified Kneser-Ney discounting. We perform pairwise ranking optimization [Hopkins and May2011] to tune feature weights. The translation quality is evaluated by case-insensitive IBM BLEU-4 metric.

The baseline translation models are generated by Moses with default parameter settings. In the contrast experiments, our DAE and HCDAE features are appended as extra features to the phrase table. The details of the used network structure in our experiments are shown in Table 1.

Table 2 presents the main translation results. We use DBN, DAE and HCDAE (with 6 layers’ network depth), input features $X_1$ and $X$, to learn 2-, 4- and 8-dimensional features, respectively. From the results, we can get some clear trends:

1. Adding new DNN features as extra features significantly improves translation accuracy (row 2-17 vs. 1), with the highest increase of 2.45 (IWSLT) and 1.52 (NIST) (row 14 vs. 1) BLEU points over the baseline features.

2. Compared with the unsupervised DBN features, our semi-supervised DAE features are more effective for translation decoder (row 3 vs. 2; row 5 vs. 4; row 8 vs. 7; row 11 vs. 10; row 13 vs. 12; row 16 vs. 15). Specially, Table 3 shows the variance distributions of the learned each dimensional DBN and DAE feature, our DAE features have bigger variance distributions which means our DAE features have more discriminative power, and also their variance distributions are more stable.

3. HCDAE outperforms single DAE for high dimensional feature learning (row 6 vs. 5; row 9 vs. 8; row 14 vs. 13; row 17 vs. 16), and further improve the performance of DAE feature learning, which can also somewhat address the bring shortcoming of the limited input features.

4. Except for the phrase feature $X_1$ [Maskey and Zhou2012], our introduced input features $X$ significantly improve the DAE feature learning (row 11 vs. 3; row 13 vs. 5; row 16 vs. 8). Specially, Table 4 shows the detailed effectiveness of our introduced input features for DAE feature learning, and the results show that each type of features are very effective for DAE feature learning.

5. Adding the original features ($X_2$, $X_3$, $X_4$ and $X_5$) and DAE/HCDAE features together can further improve translation performance (row 19-23 vs. 18), with the highest increase of 3.16 (IWSLT) and 2.06 (NIST) (row 21 vs. 1) BLEU points over the baseline features. DAE and HCDAE features are learned from the non-linear combination of the original features, they strong capture high-order correlations between the activities of the original features, as to be further interpreted to reach their potential for SMT. We suspect these learned features are complementary to the original features.

Figure 4 shows our DAE features are not only more effective but also more stable than DBN features with various network structures. Also, adding more input features ($X$ vs. $X_1$) not only significantly improves the performance of DAE feature learning, but also slightly improves the performance of DBN feature learning.

Figure 5 shows there is little change in the performance of using single DAE to learn different dimensional DAE features, but the 4-dimensional features work more better and more stable. HCDAE outperforms the single DAE and learns high-dimensional representation more effectively, especially for the peak point in each condition.

Figures 5 also shows the best network depth for DAE feature learning is 6 layers. When the network depth of DBN is 7 layers, the network depth of corresponding DAE during the fine-tuning is 13 layers. Although we have pre-trained the corresponding DBN, this DAE network is so deep, the fine-tuning does not work very well and typically finds poor local minima. We suspect this leads to the decreased performance.