Metaphor Detection with Cross-Lingual Model Transfer

To make classification decisions, we use a random forest classifier [3], an ensemble of decision tree classifiers learned from many independent subsamples of the training data. Given an input, each tree classifier assigns a probability to each label; those probabilities are averaged to compute the probability distribution across the ensemble. Random forest ensembles are particularly suitable for our resource-scarce scenario: rather than overfitting, they produce a limiting value of the generalization error as the number of trees increases,⁸⁸See Theorem 1.2 in [3] for details. and no hyperparameter tuning is required. In addition, decision-tree classifiers learn non-linear responses to inputs and often outperform logistic regression [26].⁹⁹In our experiments, random forests model slightly outperformed logistic regression and SVM classifiers. Our random forest classifier models the probability that the input syntactic relation is metaphorical. If this probability is above a threshold, the relation is classified as metaphoric, otherwise it is literal. We used the scikit-learn toolkit to train our classifiers [25].

For languages other than English, feature vectors are projected to English features using translation dictionaries. We used the Babylon dictionary,¹⁶¹⁶http://www.babylon.com which is a proprietary resource, but any bilingual dictionary can in principle be used. For a non-English word in a source language, we first obtain all translations into English. Then, we average all feature vectors related to these translations. Consider an example related to projection of WordNet supersenses. A Russian word голова is translated as head and brain. Hence, we select all the synsets of the nouns head and brain. There are 38 such synsets (33 for head and 5 for brain). Four of these synsets are associated with the supersense noun.body. Therefore, the value of the feature noun.body is $4/38\approx 0.11$.

To train an SVO metaphor classifier, we employ the TroFi (Trope Finder) dataset.¹⁷¹⁷http://www.cs.sfu.ca/~anoop/students/jbirke/ TroFi includes 3,737 manually annotated English sentences from the Wall Street Journal [2]. Each sentence contains either literal or metaphorical use for one of 50 English verbs. First, we use a dependency parser [19] to extract subject-verb-object (SVO) relations. Then, we filter extracted relations to eliminate parsing-related errors, and relations with verbs which are not in the TroFi verb list. After filtering, there are 953 metaphorical and 656 literal SVO relations which we use as a training set.

In the case of AN relations, we construct and make publicly available a training set containing 884 metaphorical AN pairs and 884 pairs with literal meaning. It was collected by two annotators using public resources (collections of metaphors on the web). At least one additional person carefully examined and culled the collected metaphors, by removing duplicates, weak metaphors, and metaphorical phrases (such as drowning students) whose interpretation depends on the context.

We collect and annotate metaphoric and literal test sentences in four languages. Thus, we compile eight test datasets, four for SVO relations, and four for AN relations. Each dataset has an equal number of metaphors and non-metaphors, i.e., the datasets are balanced. English (en) and Russian (ru) datasets have been compiled by our team and are publicly available. Spanish (es) and Farsi (fa) datasets are published elsewhere [18]. Table 1 lists test set sizes.

We used the following procedure to compile the en and ru test sets. A moderator started with seed lists of 1000 most common verbs and adjectives.¹⁸¹⁸Selection of 1000 most common verbs and adjectives achieves much broader lexical and domain coverage than what can be realistically obtained from continuous text. Our test sentence domains are, therefore, diverse: economic, political, sports, etc.

Then she used the SketchEngine, which provides searching capability for the TenTen Web corpus,¹⁹¹⁹http://trac.sketchengine.co.uk/wiki/Corpora/enTenTen to extract sentences with words that frequently co-occurred with words from the seed lists. From these sentences, she removed sentences that contained more than one metaphor, and sentences with non-SVO and non-AN metaphors. Remaining sentences were annotated by several native speakers (five for English and six for Russian), who judged AN and SVO phrases in context. The annotation instructions were general: “Please, mark in bold all words that, in your opinion, are used non-literally in the following sentences. In many sentences, all the words may be used literally.” The Fleiss’ Kappas for 5 English and 6 Russian annotators are: en-an = .76, ru-an = .85, en-svo = .75, ru-svo = .78. For the final selection, we filtered out low-agreement (.8) sentences.

The test candidate sentences were selected by a person who did not participate in the selection of the training samples. No English annotators of the test set, and only one Russian annotator out of 6 participated in the selection of the training samples. Thus, we trust that annotator judgments were not biased towards the cases that the system is trained to process.

Our task, as defined in Section 2, is to classify SVO and AN relations as either metaphoric or literal. We first conduct a 10-fold cross-validation experiment on the training set defined in Section 4.1. We represent each candidate relation using the features described in Section 3.2, and evaluate performance of the three feature categories and their combinations. This is done by computing an accuracy in the 10-fold cross validation. Experimental results are given in Table 2, where we also provide the number of features in each feature set.

These results show superior performance over previous state-of-the-art results, confirming our hypothesis that conceptual features are effective in metaphor classification. For the SVO task, the cross-validation accuracy is about 10% better than that of Tsvetkov et al. (2013). For the AN task, the cross validation accuracy is better by 8% than the result of Turney et al. (2011) (two baseline methods are described in Section 5.2). We can see that all types of features have good performance on their own (VSM is the strongest feature type). Noun supersense features alone allows us to achieve an accuracy of 75%, i.e., adjective supersense features contribute 4% to adjective-noun supersense feature combination. Experiments with the pairs of features yield better results than individual features, implying that the feature categories are not redundant. Yet, combining all features leads to even higher accuracy during cross-validation. In the case of the AN task, a difference between the All feature combination and any other combination of features listed in Table 2 is statistically significant ( for both the sign and the permutation test).

Although the first experiment shows very high scores, the 10-fold cross-validation cannot fully reflect the generality of the model, because all folds are parts of the same corpus. They are collected by the same human judges and belong to the same domain. Therefore, experiments on out-of-domain data are crucial. We carry out such experiments using held-out SVO and AN en test sets, described in Section 4.2 and Table 1. In this experiment, we measure the $f$-score. We classify SVO and AN relations using a classifier trained on the All feature combination and balanced thresholds. The values of the $f$-score are 0.76, both for SVO and AN tasks. This out-of-domain experiment suggests that our classifier is portable across domains and genres.

However, (1) different application may have different requirements for recall/precision, and (2) classification results may be skewed towards having high precision and low recall (or vice versa). It is possible to trade precision for recall by choosing a different threshold. Thus, in addition to giving a single $f$-score value for balanced thresholds, we present a Receiver Operator Characteristic (ROC) curve, where we plot a fraction of true positives against the fraction of false positives for 100 threshold values in the range from zero to one. The area under the ROC curve (AUC) can be interpreted as the probability that a classifier will assign a higher score to a randomly chosen positive example than to a randomly chosen negative example.²⁰²⁰Assuming that positive examples are labeled by ones, and negative examples are labeled by zeros. For a randomly guessing classifier, the ROC curve is a dashed diagonal line. A bad classifier has an ROC curve that goes close to the dashed diagonal or even below it.

According to ROC plots in Figure 3, all three feature sets are effective, both for SVO and for AN tasks. Abstractness and Imageability features work better for adjectives and nouns, which is in line with previous findings [40, 4]. It can be also seen that VSM features are very effective. This is in line with results of Hovy et al. (2013), who found that it is hard to improve over the classifier that uses only VSM features.

In this section, we compare our method to state-of-the-art methods of Tsvetkov et al. (2013) and of Turney et al. (2011), who focused on classifying SVO and AN relations, respectively.

In the case of SVO relations, we use software and datasets from Tsvetkov et al. (2013). These datasets, denoted as an svo-baseline, consist of 98 English and 149 Russian sentences. We train SVO metaphor detection tools on SVO relations extracted from TroFi sentences and evaluate them on the svo-baseline dataset. We also use the same thresholds for classifier posterior probabilities as Tsvetkov et al. (2013). Our approach is different from that of Tsvetkov et al. (2013) in that it uses additional features (vector space word representations) and a different classification method (we use random forests while Tsvetkov et al. (2013) use logistic regression). According to Table 3, we obtain higher performance scores for both Russian and English.

In the case of AN relations, we use the dataset (denoted as an an-baseline) created by Turney et al. (2011) (see Section 4.1 in the referred paper for details). Turney et al. (2011) manually annotated 100 pairs where an adjective was one of the following: dark, deep, hard, sweet, and worm. The pairs were presented to five human judges who rated each pair on a scale from 1 (very literal/denotative) to 4 (very non-literal/connotative). Turney et al. (2011) train logistic-regression employing only abstractness ratings as features. Performance of the method was evaluated using the 10-fold cross-validation separately for each judge.

We replicate the above described evaluation procedure of Turney et al. (2011) using their model and features. In our classifier, we use the All feature combination and the balanced threshold as described in Section 5.1.

According to results in Table 4, almost all of the judge-specific $f$-scores are slightly higher for our system, as well as the overall average $f$-score.

In both baseline comparisons, we obtain performance at least as good as in previously published studies.

In the next experiment we corroborate the main hypothesis of this paper: a model trained on English data can be successfully applied to other languages. Namely, we use a trained English model discussed in Section 5.1 to classify literal and metaphoric SVO and AN relations in English, Spanish, Farsi and Russian test sets, listed in Section 4.2. This time we used all available features.

Experimental results for all four languages, are given in Figure 6. The ROC curves for SVO and AN tasks are plotted in Figure 6 and Figure 6, respectively. Each curve corresponds to a test set described in Table 1. In addition, we perform an oracle experiment, to obtain actual $f$-score values for best thresholds. Detailed results are shown in Table 5.

Consistent results with high $f$-scores are obtained across all four languages. Note that higher scores are obtained for the Russian test set. We hypothesize that this happens due to a higher-quality translation dictionary (which allows a more accurate model transfer). Relatively lower (yet reasonable) results for Farsi can be explained by a smaller size of the bilingual dictionary (thus, fewer feature projections can be obtained). Also note that, in our experience, most of Farsi metaphors are adjective-noun constructions. This is why the AN fa dataset in Table 1 is significantly larger than SVO fa. In that, for the AN Farsi task we observe high performance scores.

Figure 6 and Table 5 confirm, that we obtain similar, robust results on four very different languages, using the same English classifiers. We view this result as a strong evidence of language-independent nature of our metaphor detection method. In particular, this shows that proposed conceptual features can be used to detect selectional preferences violation across languages.

To summarize the experimental section, our metaphor detection approach obtains state-of-the-art performance in English, is effective when applied to out-of-domain English data, and works cross-lingually.

Manual data analysis on adjective-noun pairs supports an abstractness-concreteness hypothesis formulated by several independent research studies. For example, in English we classify as metaphoric dirty word and cloudy future. Word pairs dirty diaper and cloudy weather have same adjectives. Yet they are classified as literal. Indeed, diaper is a more concrete term than word and weather is more concrete than future. Same pattern is observed in non-English datasets. In Russian, больное общество “sick society” and пустой звук “empty sound” are classified as metaphoric, while больная бабушка “sick grandmother” and пустая чашка “empty cup” are classified as literal. Spanish example of an adjective-noun metaphor is a well-known músculo económico “economic muscle”. We also observe that non-metaphoric adjective noun pairs tend to have more imageable adjectives, such as literal derecho humano “human right”. In Spanish, human is more imageable than economic.

Verb-based examples that are correctly classified by our model are: blunder escaped notice\normal@char~(metaphoric) and prisoner escaped jail\normal@char~(literal). We hypothesize that supersense features are instrumental in the correct classification of these examples: noun.person,verb.motion$\mathrm{>}$ is usually used literally, while noun.act,verb.motion$\mathrm{>}$ is used metaphorically.