Cross-lingual Model Transfer Using Feature Representation Projection

The approach proposed here, which we will refer to as feature representation projection (FRP), constitutes an alternative to direct model transfer and annotation projection and can be seen as a compromise between the two.

It is similar to direct transfer in that we also use a shared feature representation. Instead of designing this representation manually, however, we create compact monolingual feature representations for source and target languages separately and automatically estimate the mapping between the two from parallel data. This allows us to make use of language-specific annotations and account for the interplay between different types of information. For example, a certain preposition attached to a token in the source language might map into a morphological tag in the target language, which would be hard to handle for traditional direct model transfer other than using some kind of refinement procedure involving parallel data. Note also that any such refinement procedure applicable to direct transfer would likely work for FRP as well.

Compared to annotation projection, our approach may be expected to be less sensitive to parallel data quality, since we do not have to commit to a particular prediction on a given instance from parallel data. We also believe that FRP may profit from using other sources of information about the correspondence between source and target feature representations, such as dictionary entries, and thus have an edge over annotation projection in those cases where the amount of parallel data available is limited.

We consider a pair of languages $\left(L^s,L^t\right)$ and assume that an annotated training set $D_T^s=\left\{\left(x^s,y^s\right)\right\}$ is available in the source language as well as a parallel corpus of instance pairs $D^{st}=\left\{\left(x^s,x^t\right)\right\}$ and a target dataset $D_E^t=\left\{x^t\right\}$ that needs to be labeled.

We design a pair of intermediate compact monolingual feature representations ${\omega }_1^s$ and ${\omega }_1^t$ and models $M_s$ and $M_t$ to map source and target samples $x^s$ and $x^t$ from their original representations, ${\omega }_0^s$ and ${\omega }_0^t$, to the new ones. We use the parallel instances in the new feature representation

to determine the mapping $M_{ts}$ (usually, linear) between the two spaces:

Then a classification model $M_y$ is trained on the source training data

and the labels are assigned to the target samples $x^t\in D_E^t$ using a composition of the models:

Our objective is to make the feature representation sufficiently compact that the mapping between source and target feature spaces could be reliably estimated from a limited amount of parallel data, while preserving, insofar as possible, the information relevant for classification.

Estimating the mapping directly from raw categorical features (${\omega }_0$) is both computationally expensive and likely inaccurate – using one-hot encoding the feature vectors in our experiments would have tens of thousands of components. There is a number of ways to make this representation more compact. To start with, we replace word types with corresponding neural language model representations estimated using the skip-gram model []. This corresponds to $M_s$ and $M_t$ above and reduces the dimension of the feature space, making direct estimation of the mapping practical. We will refer to this representation as ${\omega }_1$.

To go further, one can, for example, apply dimensionality reduction techniques to obtain a more compact representation of ${\omega }_1$ by eliminating redundancy or define auxiliary tasks and produce a vector representation useful for those tasks. In source language, one can even directly tune an intermediate representation for the target problem.

As mentioned above we compare the performance of this approach to that of direct transfer and annotation projection. Both baselines are using the same set of features as the proposed model, as described earlier.

The shared feature representation for direct transfer is derived from ${\omega }_0$ by replacing language-specific part-of-speech tags with universal ones [] and adding cross-lingual word clusters [] to word types. The word types themselves are left as they are in the source language and replaced with their gloss translations in the target one []. In English-Czech and Czech-English we also use the dependency relation information, since the annotations are partly compatible.

The annotation projection baseline implementation is straightforward. The source-side instances from a parallel corpus are labeled using a classifier trained on source-language training data and transferred to the target side. The resulting annotations are then used to train a target-side classifier for evaluation. Note that predicate and argument identification in both languages is performed using monolingual classifiers and only aligned pairs are used in projection. A more common approach would be to project the whole structure from the source language, but in our case this may give unfair advantage to feature representation projection, which relies on target-side argument identification.

We use the same type of log-linear classifiers in the model itself and the two baselines to avoid any discrepancy due to learning procedure. These classifiers are implemented using Pylearn2 [], based on Theano []. We also use this framework to estimate the linear mapping $M_{ts}$ between source and target feature spaces in FRP.

The 250-dimensional word representations for ${\omega }_1$ are obtained using Word2vec tool. Both monolingual data and that from the parallel corpus are included in the training. In  the authors consider embeddings of up to 800 dimensions, but we would not expect to benefit as much from larger vectors since we are using a much smaller corpus to train them. We did not tune the size of the word representation to our task, as this would not be appropriate in a cross-lingual transfer setup, but we observe that the classifier is relatively robust to their dimension when evaluated on source language – in our experiments the performance of the monolingual classifier does not improve significantly if the dimension is increased past 300 and decreases only by a small margin (less than one absolute point) if it is reduced to 100. It should be noted, however, that the dimension that is optimal in this sense is not necessarily the best choice for FRP, especially if the amount of available parallel data is limited.

We use two language pairs for evaluation: English-Czech and English-French. In the first case, the data is converted from Prague Czech-English Dependency Treebank 2.0 [] using the script from . In the second, we use CoNLL 2009 shared task [] corpus for English and the manually corrected dataset from for French. Since the size of the latter dataset is relatively small – one thousand sentences – we reserve the whole dataset for testing and only evaluate transfer from English to French, but not the other way around. Datasets for other languages are sufficiently large, so we take 30 thousand samples for testing and use the rest as training data. The validation set in each experiment is withheld from the corresponding training corpus and contains 10 thousand samples.

Parallel data for both language pairs is derived from Europarl [], which we pre-process using mate-tools [].