Predicting the relevance of distributional semantic similarity with contextual information

We use a distributional resource for French, built on a 200M word corpus extracted from the French Wikipedia, following principles laid out in [] from a structured model [], i.e. using syntactic contexts. In this approach, contexts are triples (governor,relation,dependent) derived from syntactic dependency structures. Governors and dependents are verbs, adjectives and nouns. Multiword units are available, but they form a very small subset of the resulting neighbours. Base elements in the thesaurus are of two types: arguments (dependents’ lemma) and predicates (governor+relation). This is to keep the predicate/argument distinction since similarities will be computed between predicate pairs or argument pairs, and a lexical item can appear in many predicates and as an argument (e.g. interest as argument, interest_for as one predicate). The similarity of distributions was computed with Lin’s score [].

We will talk of lexical neighbours or distributional neighbours to label pairs of predicates or arguments, and in the rest of the paper we consider only lexical pairs with a Lin score of at least 0.1, which means about 1.4M pairs. This somewhat arbitrary level is an a priori threshold to limit the resulting database, and it is conservative enough not to exclude potential interesting relations. The distribution of scores is given figure 1; 97% of the selected pairs have a score between 0.1 and 0.29.

To ease the use of lexical neighbours in our experiments, we merged together predicates that include the same lexical unit, a posteriori. Thus there is no need for a syntactic analysis of the context considered when exploiting the resource, and sparsity is less of an issue¹¹Whenever two predicates with the same lemma have common neighbours, we average the score of the pairs..

In order to evaluate the resource, we set up an annotation in context: pairs of lexical items are to be judged in their context of use, in texts where they occur together. To verify that this methodology is useful, we did a preliminary annotation to contrast judgment on lexical pairs with or without this contextual information. Then we made a larger annotation in context once we were assured of the reliability of the methodology.

For the preliminary test, we asked three annotators to judge the similarity of pairs of lexical items without any context (no-context), and to judge the similarity of pairs presented within a paragraph where they both occur (in context). The three annotators were linguists, and two of them (1 and 3) knew about the resource and how it was built. For each annotation, $100$ pairs were randomly selected, with the following constraints:

• •

  for the no-context annotation, candidate pairs had a Lin score above $0.2$, which placed them in the top $14\mathrm{\%}$ of lexical neighbours with respect to the similarity level.

• •

  for the in context annotation, the only constraint was that the pairs occur in the same paragraph somewhere in the corpus used to build the resource. The example paragraph was chosen at random.

The guidelines given in both cases were the same: “Do you think the two words are semantically close ? In other words, is there a semantic relation between them, either classical (synonymy, hypernymy, co-hyponymy, meronymy, co-meronymy) or not (the relation can be paraphrased but does not belong to the previous cases) ?”

For the pre-test, agreement was rather moderate without context (the average of pairwise kappas was .46), and much better with a context (average = .68), with agreement rates above 90%. This seems to validate the feasability of a reliable annotation of relatedness in context, so we went on for a larger annotation with two of the previous annotators.

For the larger annotation, the protocol was slightly changed: two annotators were given 42 full texts from the original corpus where lexical neighbours occurred. They were asked to judge the relation between two items types, regardless of the number of occurrences in the text. This time there was no filtering of the lexical pairs beyond the 0.1 threshold of the original resource. We followed the well-known postulate [] that all occurrences of a word in the same discourse tend to have the same sense (“one sense per discourse”), in order to decrease the annotator workload. We also assumed that the relation between these items remain stable within the document, an arguably strong hypothesis that needed to be checked against inter-annotator agreement before beginning the final annotation . It turns out that the kappa score ($0.80$) shows a better inter-annotator agreement than during the preliminary test, which can be explained by the larger context given to the annotator (the whole text), and thus more occurrences of each element in the pair to judge, and also because the annotators were more experienced after the preliminary test. Agreement measures are summed-up table 1. An excerpt of an example text, as it was presented to the annotators, is shown figure 2.

Overall, it took only a few days to annotate 9885 pairs of lexical items. Among the pairs that were presented to the annotators, about 11% were judged as relevant by the annotators. It is not easy to decide if the non-relevant pairs are just noise, or context-dependent associations that were not present in the actual text considered (for polysemy reasons for instance), or just low-level associations. An important aspect is thus to guarantee that there is a correlation between the similarity score (Lin’s score here), and the evaluated relevance of the neighbour pairs. Pearson correlation factor shows that Lin score is indeed significantly correlated to the annotated relevance of lexical pairs, albeit not strongly ($r=0.159$).

The produced annotation²²Freely available, and distributed with this submission. can be used as a reference to explore various aspects of distributional resources, with the caveat that it is as such a bit dependent on the particular resource used. We nonetheless assume that some of the relevant pairs would appear in other thesauri, or would be of interest in an evaluation of another resource.

The first thing we can analyse from the annotated data is the impact of a threshold on Lin’s score to select relevant lexical pairs. The resource itself is built by choosing a cut-off which is supposed to keep pairs with a satisfactory similarity, but this threshold is rather arbitrary. Figure 3 shows the influence of the threshold value to select relevant pairs, when considering precision and recall of the pairs that are kept when choosing the threshold, evaluated against the human annotation of relevance in context. In case one wants to optimize the F-score (the harmonic mean of precision and recall) when extracting relevant pairs, we can see that the optimal point is at .24 for a threshold of .22 on Lin’s score. This can be considered as a baseline for extraction of relevant lexical pairs, to which we turn in the following section.

For each pair neighbour${}_a$/neighbour${}_b$, we computed a set of features from Wikipedia (the corpus used to derive the distributional similarity): We first computed the frequencies of each item in the corpus, $fre{q}_a$ and $fre{q}_b$, from which we derive

• •

  $fre{q}_{min}$, $fre{q}_{max}$ : the min and max of $fre{q}_a$ and $fre{q}_b$ ;

• •

  $fre{q}_{\times }$: the combination of the two, or $\log \left(fre{q}_a\times fre{q}_b\right)$

We also measured the syntagmatic association of neighbour${}_a$ and neighbour${}_b$, with a mutual information measure [], computed from the cooccurrence of two tokens within the same paragraph in Wikipedia. This is a rather large window, and thus gives a good coverage with respect to the neighbour database (70% of all pairs).

A straightforward parameter to include to predict the relevance of a link is of course the similarity measure itself, here Lin’s information measure. But this can be complemented by additional information on the similarity of the neighbours, namely:

• •

  each neighbour productivity : $pro{d}_a$ and $pro{d}_b$ are defined as the numbers of neighbours of respectively neighbour${}_a$ and neighbour${}_b$ in the database (thus related tokens with a similarity above the threshold), from which we derive three features as for frequencies: the min, the max, and the log of the product. The idea is that neighbours whith very high productivity give rise to less reliable relations.

• •

  the ranks of tokens in other related items neighbours: $ran{k}_{a-b}$ is defined as the rank of neighbour${}_a$ among neighbours of neighbour${}_b$  ordered with respect to Lin’s score; $ran{g}_{b-a}$ is defined similarly and again we consider as features the min, max and log-product of these ranks.

We add two categorial features, of a more linguistic nature:

• •

  $cats$ is the pair of part-of-speech for the related items, e.g. to distinguish the relevance of NN or VV pairs.

• •

  $predarg$ is related to the predicate/argument distinction: are the related items predicates or arguments ?

The last set of features derive from the occurrences of related tokens in the considered discourses:

First, we take into account the frequencies of items within the text, with three features as before: the min of the frequencies of the two related items, the max, and the log-product. Then we consider a tf$\mathrm{\cdot }$idf [] measure, to evaluate the specificity and arguably the importance of a word in a document or within a document. Several variants of tf$\mathrm{\cdot }$idf have been proposed to adapt the measure to more local areas in a text with respect to the whole document. For instance [] propose a tf$\mathrm{\cdot }$isf(term frequency $\mathrm{\cdot }$ inverse sentence frequency), for topic segmentation. We similarly defined a tf$\mathrm{\cdot }$ipfmeasure based on the frequency of a word within a paragraph with respect to its frequency within the text. The resulting feature we used is the product of this measure for neighbour${}_a$ and neighbour${}_b$.

A few other contextual features are included in the model: the distances between pairs of related items, instantiated as:

• •

  distance in words between occurrences of related word types:

  • –

    minimal distance between two occurrences ($sd$)

  • –

    maximal distance between two occurrences ($gd$)

  • –

    average distance ($ad$) ;

• •

  boolean features indicating whether neighbour${}_a$ and neighbour${}_b$ appear in the same sentence ($cop{r}_s$) or the same paragraph ($cop{r}_{para}$).

Finally, we took into account the network of related lexical items, by considering the largest sets of words present in the text and connected in the database (self-connected components), by adding the following features:

• •

  the degree of each lemma, seen as a node in this similarity graph, combined as above in minimal degree of the pair, maximal degree, and product of degrees ($prodtx{t}_{min}$, $prodtx{t}_{max}$, $prodtx{t}_{\times }$). This is the number of pairs (present in the text) where a lemma appears in.

• •

  a boolean feature $cc$ saying whether a lexical pair belongs to a connected component of the text, except the largest. This reflects the fact that a small component may concern a lexical field which is more specific and thus more relevant to the text.

  Figure 4 shows examples of self-connected components in an excerpt of the page on Gorille (gorilla), e.g. the set {pelage, dos, fourrure} (coat, back, fur).

  Le gorille est après le bonobo et le chimpanzé , du de , l’ le plus de l’ . Cette a été par les entre les et les . Notre ne que de 2 % de celui du gorille .

  Redressés , les gorilles une de 1,75 , mais ils sont en un peu plus car ils ont les fléchis . L’ des la du et 2,75 .

  Il une de entre les : les de 90 à 150 et les jusqu’ à 275. En , particulièrement bien , ils 350 .

  Le du et de l’ . Chez les les plus se sur le une argenté , d’ où leur de “ argentés” . Le des gorilles de est particulièrement et soyeux .

  Comme tous les anthropoïdes , les gorilles sont dépourvus de . Leur est , le et les sont glabres et ils des torus supra-orbitaires marqués .

  Figure 4: A few connected lexical components of the similarity graph, projected on a text, each in a different color. The groups are, in order of appearance of the first element: {genetic, close, human}, {similarity, kinship}, {chromosome, genome}, {male, female}, {coat, back, fur}, {age/N, aged/A}, {ear, tail, face}. The text describes the gorilla species, more particularly its morphology. Gray words are other lexical elements in the neighbour database.

The last feature is probably not entirely independent from the productivity of an item, or from the tf.ipf measure.

Table 2 sums up the features used in our model.

Our task is to identify relevant similarities between lexical items, between all possible related pairs, and we want to train an inductive model, a classifier, to extract the relevant links. We have seen that the relevant/not relevant classification is very imbalanced, biased towards the “not relevant” category (about 11%/89%), so we applied methods dedicated to counter-balance this, and will focus on the precision and recall of the predicted relevant links.

Following a classical methodology, we made a 10-fold cross-validation to evaluate robustly the performance of the classifiers. We tested a few popular machine learning methods, and report on two of them, a naive bayes model and the best method on our dataset, the Random Forest classifier []. Other popular methods (maximum entropy, SVM) have shown slightly inferior combined F-score, even though precision and recall might yield more important variations. As a baseline, we can also consider a simple threshold on the lexical similarity score, in our case Lin’s measure, which we have shown to yield the best F-score of 24% when set at 0.22.

To address class imbalance, two broad types of methods can be applied to help the model focus on the minority class. The first one is to resample the training data to balance the two classes, the second one is to penalize differently the two classes during training when the model makes a mistake (a mistake on the minority class being made more costly than on the majority class). We tested the two strategies, by applying the classical Smote method of [] as a kind of resampling, and the ensemble method MetaCost of [] as a cost-aware learning method. Smote synthetizes and adds new instances similar to the minority class instances and is more efficient than a mere resampling. MetaCost is an interesting meta-learner that can use any classifier as a base classifier. We used Weka’s implementations of these methods [], and our experiments and comparisons are thus easily replicated on our dataset, provided with this paper, even though they can be improved by refinements of these techniques. We chose the following settings for the different models: naive bayes uses a kernel density estimation for numerical features, as this generally improves performance. For Random Forests, we chose to have ten trees, and each decision is taken on a randomly chosen set of five features. For resampling, Smote advises to double the number of instances of the minority class, and we observed that a bigger resampling degrades performances. For cost-aware learning, a sensible choice is to invert the class ratio for the cost ratio, i.e. here the cost of a mistake on a relevant link (false negative) is exactly 8.5 times higher than the cost on a non-relevant link (false positive), as non-relevant instances are 8.5 times more present than relevant ones.

We are interested in the precision and recall for the “relevant” class. If we take the best simple classifier (random forests), the precision and recall are $68.1\mathrm{\%}$ and $24.2\mathrm{\%}$ for an F-score of $35.7\mathrm{\%}$, and this is significantly beaten by the Naive Bayes method as precision and recall are more even (F-score of 41.5%). This is already a big improvement on the use of the similarity measure alone (24%). Also note that predicting every link as relevant would result in a 2.6% precision, and thus a 5% F-score. The random forest model is significantly improved by the balancing techniques: the overall best F-score of 46.3% is reached with Random Forests and the cost-aware learning method. Table 3 sums up the scores for the different configurations, with precision, recall, F-score and the confidence interval on the F-score. We analysed the learning curve by doing a cross-validation on reduced set of instances (from 10% to 90%); F1-scores range from 37.3% with 10% of instances and stabilize at 80%, with small increment in every case.

The filtering approach we propose seems to yield good results, by augmenting the similarity built on the whole corpus with signals from the local contexts and documents where related lexical items appear together.

To try to analyse the role of each set of features, we repeated the experiment but changed the set of features used during training, and results are shown table 4 for the best method (RF with cost-aware learning).

We can see that similarity-related features (measures, ranks) have the biggest impact, but the other ones also seem to play a significant role. We can draw the tentative conclusion that the quality of distributional relations depends on the contextualizing of the related lexical items, beyond just the similarity score and the ranks of items as neighbours of other items.