Looking at Unbalanced Specialized Comparable Corpora
for Bilingual Lexicon Extraction

The main work in bilingual lexicon extraction from comparable corpora is based on lexical context analysis and relies on the simple observation that a word and its translation tend to appear in the same lexical contexts. The basis of this observation consists in the identification of “first-order affinities” for each source and target language: “First-order affinities describe what other words are likely to be found in the immediate vicinity of a given word” [14, p. 279]. These affinities can be represented by context vectors, and each vector element represents a word which occurs within the window of the word to be translated (e.g. a seven-word window approximates syntactic dependencies). In order to emphasize significant words in the context vector and to reduce word-frequency effects, the context vectors are normalized according to an association measure. Then, the translation is obtained by comparing the source context vector to each translation candidate vector after having translated each element of the source vector with a general dictionary.

The implementation of the standard approach can be carried out by applying the following three steps [29, 3, 7, 22, 18, among others]:

Computing context vectors

We collect all the words in the context of each word $i$ and count their occurrence frequency in a window of $n$ words around $i$. For each word $i$ of the source and the target languages, we obtain a context vector $v_i$ which gathers the set of co-occurrence words $j$ associated with the number of times that $j$ and $i$ occur together $cooc\left(i,j\right)$. In order to identify specific words in the lexical context and to reduce word-frequency effects, we normalize context vectors using an association score such as Mutual Information, Log-likelihood, or the discounted log-odds (LO) [8] (see equation 1 and Table 1 where $N=a+b+c+d$).

Transferring context vectors

Using a bilingual dictionary, we translate the elements of the source context vector. If the bilingual dictionary provides several translations for an element, we consider all of them but weight the different translations according to their frequency in the target language.

Finding candidate translations

For a word to be translated, we compute the similarity between the translated context vector and all target vectors through vector distance measures such as Jaccard or Cosine (see equation 2 where $asso{c}_j^i$ stands for “association score”, $v_k$ is the transferred context vector of the word $k$ to translate, and $v_l$ is the context vector of the word $l$ in the target language). Finally, the candidate translations of a word are the target words ranked following the similarity score.

This approach is sensitive to the choice of parameters such as the size of the context, the choice of the association and similarity measures. The most complete study about the influence of these parameters on the quality of word alignment has been carried out by Laroche and Langlais [18].

The standard approach is used by most researchers so far [28, 12, 25, 29, 3, 7, 13, 22, 18, 26, 2, among others] with the implicit hypothesis that comparable corpora are balanced. As McEnery and Xiao (2007, p. 21) observe, a specialized comparable corpus is built as balanced by analogy with a parallel corpus: “Therefore, in relation to parallel corpora, it is more likely for comparable corpora to be designed as general balanced corpora.”. For instance, Table 2 describes the comparable corpora used in the main work dedicated to bilingual lexicon extraction for which the ratio between the size of the source and the target texts is comprised between 1 and 1.8.

In fact, the assumption that words which have the same meaning in different languages should have the same lexical context distributions does not involve working with balanced comparable corpora. To our knowledge, no attention¹¹We only found mention of this aspect in Diab and Finch (2000, p. 1501) “In principle, we do not have to have the same size corpora in order for the approach to work”. has been paid to the problem of using unbalanced comparable corpora for bilingual lexicon extraction. Since the context vectors are computed from each part of the comparable corpus rather than through the parts of the comparable corpora, the standard approach is relatively insensitive to differences in corpus sizes. The only precaution for using the standard approach with unbalanced corpora is to normalize the association measure (for instance, this can be done by dividing each entry of a given context vector by the sum of its association scores).

Since comparable corpora are usually small in specialized domains (see Table 2), the discriminative power of context vectors (i.e. the observations of word co-occurrences) is reduced. One way to deal with this problem is to re-estimate co-occurrence counts by a prediction function [15]. This consists in assigning to each observed co-occurrence count of a small comparable corpora, a new value learned beforehand from a large training corpus.

In order to make co-occurrence counts more discriminant and in the same way as Hazem and Morin [15], one strategy consists in addressing this problem through regression: given training corpora of small and large size (abundant in the general domain), we predict word co-occurrence counts in order to make them more reliable. We then apply the resulting regression function to each word co-occurrence count as a pre-processing step of the standard approach. Our work differs from Hazem and Morin [15] in two ways. First, while they experienced the linear regression model, we propose to contrast different regression models. Second, we apply regression to unbalanced comparable corpora and study the impact of prediction when applied to the source texts, the target texts and both source and target texts of the used comparable corpora.

We use regression analysis to describe the relationship between word co-occurrence counts in a large corpus (the response variable) and word co-occurrence counts in a small corpus (the predictor variable). As most regression models have already been described in great detail [5, 1], the derivation of most models is only briefly introduced in this work.

As we can not claim that the prediction of word co-occurrence counts is a linear problem, we consider in addition to the simple linear regression model ($Lin$), a generalized linear model which is the logistic regression model ($Logit$) and non linear regression models such as polynomial regression model ($Pol{y}^n$) of order $n$. Given an input vector $x\in {ℝ}^m$, where $x_1$,…,$x_m$ represent features, we find a prediction $\widehat{y}\in {ℝ}^m$ for the co-occurrence count of a couple of words $y\in ℝ$ using one of the regression models presented below:

where ${\beta }_i$ are the parameters to estimate.

Let us denote by $f$ the regression function and by $cooc\left(w_i,w_j\right)$ the co-occurrence count of the words $w_i$ and $w_j$. The resulting predicted value of $cooc\left(w_i,w_j\right)$, noted $\widehat{cooc}\left(w_i,w_j\right)$ is given by the following equation:

In the past few years, several contributions have been proposed to improve each step of the standard approach.

Prochasson et al. (2009) enhance the representativeness of the context vector by strengthening the context words that happen to be transliterated words and scientific compound words in the target language. Ismail and Manandhar (2010) also suggest that context vectors should be based on the most important contextually relevant words (in-domain terms), and thus propose a method for filtering the noise of the context vectors. In another way, Rubino and Linarès (2011) improve the context words based on the hypothesis that a word and its candidate translations share thematic similarities. Yu and Tsujii (2009) and Otero (2007) propose, for their part, to replace the window-based method by a syntax-based method in order to improve the representation of the lexical context.

To improve the transfer context vectors step, and increase the number of elements of translated context vectors, Chiao and Zweigenbaum (2003) and Morin and Prochasson (2011) combine a standard general language dictionary with a specialized dictionary, whereas Déjean et al. (2002) use the hierarchical properties of a specialized thesaurus. Koehn and Knight (2002) automatically induce the initial seed bilingual dictionary by using identical spelling features such as cognates and similar contexts. As regards the problem of words ambiguities, Bouamor et al. (2013) carried out word sense disambiguation process only in the target language whereas Gaussier et al. (2004) solve the problem through the source and target languages by using approaches based on CCA (Canonical Correlation Analysis) and multilingual PLSA (Probabilistic Latent Semantic Analysis).

The rank of candidate translations can be improved by integrating different heuristics. For instance, Chiao and Zweigenbaum (2002) introduce a heuristic based on word distribution symmetry. From the ranked list of candidate translations, the standard approach is applied in the reverse direction to find the source counterparts of the first target candidate translations. And then only the target candidate translations that had the initial source word among the first reverse candidate translations are kept. Laroche and Langlais (2010) suggest a heuristic based on the graphic similarity between source and target terms. Here, candidate translations which are cognates of the word to be translated are ranked first among the list of translation candidates.

For our experiments, we used two specialized French/English comparable corpora:

Breast cancer corpus

This comparable corpus is composed of documents collected from the Elsevier website²²http://www.elsevier.com. The documents were taken from the medical domain within the sub-domain of “breast cancer”. We have automatically selected the documents published between 2001 and 2008 where the title or the keywords contain the term cancer du sein in French and breast cancer in English. We collected 130 French documents (about 530,000 words) and 1,640 English documents (about 7.4 million words). We split the English documents into 14 parts each containing about 530,000 words.

Diabetes corpus

The documents making up the French part of the comparable corpus have been craweled from the web using three keywords: diabète (diabetes), alimentation (food), and obésité (obesity). After a manual selection, we only kept the documents which were relative to the medical domain. As a result, 65 French documents were extracted (about 257,000 words). The English part has been extracted from the medical website PubMed³³http://www.ncbi.nlm.nih.gov/pubmed/ using the keywords: diabetes, nutrition and feeding. We only kept the free fulltext available documents. As a result, 2,339 English documents were extracted (about 3,5 million words). We also split the English documents into 14 parts each containing about 250,000 words.

The French and English documents were then normalised through the following linguistic pre-processing steps: tokenisation, part-of-speech tagging, and lemmatisation. These steps were carried out using the TTC TermSuite⁴⁴http://code.google.com/p/ttc-project that applies the same method to several languages including French and English. Finally, the function words were removed and the words occurring less than twice in the French part and in each English part were discarded. Table 3 shows the number of distinct words (# words) after these steps. It also indicates the comparability degree in percentage (comp.) between the French part and each English part of each comparable corpus. The comparability measure [19] is based on the expectation of finding the translation for each word in the corpus and gives a good idea about how two corpora are comparable. We can notice that all the comparable corpora have a high degree of comparability with a better comparability of the breast cancer corpora as opposed to the diabetes corpora. In the remainder of this article, $[$breast cancer corpus $i]$ for instance stands for the breast cancer comparable corpus composed of the unique French part and the English part $i$ ($i\in \left[1,14\right]$).

The bilingual dictionary used in our experiments is the French/English dictionary ELRA-M0033 available from the ELRA catalogue⁵⁵http://www.elra.info/. This resource is a general language dictionary which contains only a few terms related to the medical domain.

To evaluate the quality of terminology extraction, we built a bilingual terminology reference list for each comparable corpus. We selected all French/English single words from the UMLS⁶⁶http://www.nlm.nih.gov/research/umls meta-thesaurus. We kept only i) the French single words which occur more than four times in the French part and ii) the English single words which occur more than four times in each English part $i$⁷⁷The threshold sets to four is required to build a bilingual terminology reference list composed of about a hundred words. This value is very low to obtain representative context vectors. For instance, Prochasson and Fung (2011) showed that the standard approach is not relevant for infrequent words (since the context vectors are very unrepresentative i.e. poor in information).. As a result of filtering, 169 French/English single words were extracted for the breast cancer corpus and 244 French/English single words were extracted for the diabetes corpus. It should be noted that the evaluation of terminology extraction using specialized comparable corpora often relies on lists of a small size: 95 single words in Chiao and Zweigenbaum [3], 100 in Morin et al. [22], 125 and 79 in Bouamor et al. [2].

In order to evaluate the influence of corpus size on the bilingual terminology extraction task, two experiments have been carried out using the standard approach. We first performed an experiment using each comparable corpus independently of the others (we refer to these corpora as balanced corpora). We then conducted a second experiment where we varied the size of the English part of the comparable corpus, from 530,000 to 7.4 million words for the breast cancer corpus in 530,000 words steps, and from 250,000 to 3.5 million words for the diabetes corpus in 250,000 words steps (we refer to these corpora as unbalanced corpora). In the experiments reported here, the size of the context window $w$ was set to 3 (i.e. a seven-word window that approximates syntactic dependencies), the retained association and similarity measures were the discounted log-odds and the Cosine (see Section 2.1). The results shown were those that give the best performance for the comparable corpora used individually.

Table 4 shows the results of the standard approach on the balanced and the unbalanced breast cancer and diabetes comparable corpora. Each column corresponds to the English part $i$ ($i\in \left[1,14\right]$) of a given comparable corpus. The first line presents the results for each individual comparable corpus and the second line presents the results for the cumulative comparable corpus. For instance, the column 3 indicates the MAP obtained by using a comparable corpus that is composed i) only of $[$breast cancer corpus 3$]$ (MAP of 21.0%), and ii) of $[$breast cancer corpus 1, 2 and 3$]$ (MAP of 34.7%).

As a preliminary remark, we can notice that the results differ noticeably according to the comparable corpus used individually (MAP variation between 21.0% and 29.6% for the breast cancer corpora and between 10.5% and 16.5% for the diabetes corpora). We can also note that the MAP of all the unbalanced comparable corpora is always higher than any individual comparable corpus. Overall, starting with a MAP of 26.1% as provided by the balanced $[$breast cancer corpus 1$]$, we are able to increase it to 42.3% with the unbalanced $[$breast cancer corpus 12$]$ (the variation observed for some unbalanced corpora such as $[$diabetes corpus 12, 13 and 14$]$ can be explained by the fact that adding more data in the source language increases the error rate of the translation phase of the standard approach, which leads to the introduction of additional noise in the translated context vectors).

The aim of this experiment is two-fold: first, we want to evaluate the usefulness of predicting word co-occurrence counts and second, we want to find out whether it is more appropriate to apply prediction to the source side, the target side or both sides of the bilingual comparable corpora.

In this last experiment we contrast the standard approach applied to the balanced and unbalanced corpora noted $Balanced$ and $Unbalanced$ with the standard approach combined with the prediction model noted $Balanced+Prediction$ and $Unbalanced+Prediction$.

Figure 1 illustrates the results of the experiments conducted on the breast cancer corpus. We can see that the $Unbalanced$ approach significantly outperforms the baseline ($Balanced$). The big difference between the $Balanced$ and the $Unbalanced$ approaches would indicate that the latter is optimal. We can also notice that the prediction model applied to the balanced corpus ($Balanced+Prediction$) slightly outperforms the baseline while the $Unbalanced+Prediction$ approach significantly outperforms the three other approaches (moreover the variation observed with the $Unbalanced$ approach are lower than the $Unbalanced+Prediction$ approach). Overall, the prediction increases the performance of the standard approach especially for unbalanced corpora.

The results of the experiments conducted on the diabetes corpus are shown in Figure 1. As for the previous experiment, we can see that the $Unbalanced$ approach significantly outperforms the $Balanced$ approach. This confirms the unbalanced hypothesis and would motivate the use of unbalanced corpora when they are available. We can also notice that the $Balanced+Prediction$ approach slightly outperforms the baseline while the $Unbalanced+Prediction$ approach gives the best results. Here also, the prediction increases the performance of the standard approach especially for unbalanced corpora. It is clear that in addition to the benefit of using unbalanced comparable corpora, prediction shows a positive impact on the performance of the standard approach.