Applying a Naive Bayes Similarity Measure to
Word Sense Disambiguation

Formally, given two sets $𝐞=\left\{e_i\right\}$ and $𝐟=\left\{f_j\right\}$ each consisting of multiple random events, the proposed model measures the probabilistic association $p\left(𝐟\right|𝐞)$ between $𝐞$ and $𝐟$. Under the assumption of conditional independence among the events in each set, a Naive Bayes treatment of the measure can be formulated as:

In the second expression, Bayes’s rule is applied not only to take advantage of the conditional independence among $e_i$’s, but also to facilitate probability estimation, since $p\left(\left\{e_i\right\}\right|f_j)$ is easier to estimate in the context of WSD, where sample spaces of $𝐞$ and $𝐟$ become asymmetric (Section 3.2).

In the context of WSD, $𝐞$ can be regarded as an instance of a polysemous word $w$, while $𝐟$ represents certain lexical knowledge about the sense $s$ of $w$ manifested by $𝐞$.¹¹Think of the notations $𝐞$ and $𝐟$ mnemonically as exemplars and features, respectively. WSD is thus formulated as identifying the sense $s^{*}$ in the sense inventory $𝒮$ of $w$ s.t.:

In one of their simplest forms, $e_i$’s correspond to co-occurring words in the instance of $w$, and $f_j$’s consist of the gloss words of sense $s$. Consequently, $p\left(𝐟\right|𝐞)$ is essentially measuring the association between context words of $w$ and definition texts of $s$, i.e., the gloss-context association in the simplified Lesk algorithm (). A major difference, however, is that instead of using hard, overlap counts between the two sets of words from the gloss and the context, this probabilistic treatment can implicitly model the distributional similarity among the elements $e_i$ and $f_j$ (and consequently between the sets $𝐞$ and $𝐟$) over a wider range of contexts. The result is a “softer” proxy of association than the binary view of overlaps in existing Lesk variants.

The foregoing discussion offers a second motivation for applying Bayes’s rule on the second expression in Equation (1): it is easier to estimate $p\left(e_i\right|f_j)$ than $p\left(f_j\right|e_i)$, since the vocabulary for the lexical knowledge features ($f_j$) is usually more limited than that of the contexts ($e_i$) and hence estimation of the former suffices on a smaller amount of data than that of the latter.

The input of the proposed NBM is bags of words, and thus it is straightforward to incorporate various forms of lexical knowledge (LK) for word senses: by concatenating a tokenized knowledge source to the existing knowledge representation $𝐟$, while the similarity measure remains unchanged.

The availability of LK largely depends on the sense inventory used in a WSD task. WordNet senses are often used in Senseval and SemEval tasks, and hence senses (or synsets, and possibly their corresponding word forms) that are semantic related to the inventory senses under WordNet relations are easily obtainable and have been exploited by many existing studies.

As pointed out by , however, “not all of these relations are equally helpful.” Relation pairs involving hyponyms were shown to result in better F-measure when used in gloss overlaps (). The authors attributed the phenomenon to the the multitude of hyponyms compared to other relations. We further hypothesize that, beyond sheer numbers, synonyms and hyponyms offer stronger semantic specification that helps distinguish the senses of a given ambiguous word, and thus are more effective knowledge sources for WSD.

Take the word plant for example. Selected hypernyms, hyponyms, and synonyms pertaining to its two senses factory and life form are listed in Table 1. Hypernyms can be overly general terms (e.g., being). Although conceptually helpful for humans in coarse-grained WSD, this generality is likely to inflate the hypernyms’ probabilistic estimation. Hyponyms, on the other hand, help specify their corresponding senses with information that is possibly missing from the often overly brief glosses: the many technical terms as hyponyms in Table 1 — though rare — are likely to occur in the (possibly domain-specific) contexts that are highly typical of the corresponding senses. Particularly for the NBM, the co-occurrence is likely to result in stronger gloss-definition associations when similar contexts appear in a WSD instance.

We also observe that some semantically related words appear under rare senses (e.g., still as an alcohol-manufacturing plant, and annual as a one-year-life-cycle plant; omitted from Table 1). This is a general phenomenon in gloss-based WSD and is beyond the scope of the current discussion.²²We do, however, refer curious readers to the work of for a novel treatment of a similar problem. Overall, all three sources of LK may complement each other in WSD tasks, with hyponyms particularly promising in both quantity and quality compared to hypernyms and synonyms.³³Note that LK expansion is a feature of our model rather than a requirement. What type of knowledge to include is eventually a decision made by the user based on the application and LK availability.

A most open-ended question is how to estimate the probabilities in Equation (1). In WSD in particular, the estimation concerns the marginal and conditional probabilities of and between word tokens. Many options are available to this end in statistical machine learning (MLE, MAP, etc.), information theory (), as well as the rich body of research in lexical semantic similarity ).

Here we choose maximum likelihood — not only for its simplicity, but also to demonstrate model strength with a relatively crude probability estimation. To avoid underflow, Equation (1) is estimated as the following log probability:

where $c\left(x\right)$ is the count of word $x$, $c(\cdot )$ is the corpus size, $c\left(x,y\right)$ is the joint count of $x$ and $y$, and $\left|𝐯\right|$ is the dimension of vector $𝐯$.

Nonetheless, we do investigate how model performance responds to estimation quality. Specifically in WSD, a source corpus is defined as the source of the majority of the WSD instances in a given dataset, and a baseline corpus of a smaller size and less resemblance to the instances is used for all datasets. The assumption is that a source corpus offers better estimates for the model than the baseline corpus, and difference in model performance is expected when using probability estimation of different quality.

Various aspects of the model discussed in Section 3 are evaluated in the English lexical sample tasks from Senseval-2 () and SemEval-2007 (). Training sections are used as development data and test sections held out for final testing. Model performance is evaluated in terms of WSD accuracy using Equation (2) as the scoring function. Accuracy is defined as the number of correct responses over the number of instances. Because it is a rare event for the NBM to produce identical scores,⁴⁴This has never occurred in the hundreds of thousands of runs in our development process. the model always proposes a unique answer and accuracy is thus equivalent to F-score commonly used in existing reports.

Multiword expressions (MWEs) in the Senseval-2 sense inventory are not explicitly marked in the contexts. Several of the top-ranking systems implemented their own MWE detection algorithms (). Without digressing to the details of MWE detection — and meanwhile, to ensure fair comparison with existing systems — we implement two variants of the prediction module, one completely ignorant of MWE and defaulting to INCORRECT for all MWE-related answers, while the other assuming perfect MWE detection and performing regular disambiguation algorithm on the MWE-related senses (not defaulting to CORRECT). All results reported for Senseval-2 below are harmonic means of the two outcomes.

Each inventory sense is represented by a set of LK tokens (e.g., definition texts, synonyms, etc.) from their corresponding WordNet synset (or in the coarse-grained case, a concatenation of tokens from all synsets in a sense group). The MIT-JWI library () is used for accessing WordNet. Usage examples in glosses (included by the library by default) are removed in our experiments.⁵⁵We also compared the two Lesk baselines (with and without usage examples) on the development data but did not observe significant differences as reported by .

Basic pre-processing is performed on the contexts and the glosses, including lower-casing, stopword removal, lemmatization on both datasets, and tokenization on the Senseval-2 instances.⁶⁶The SemEval-2007 instances are already tokenized. Stanford CoreNLP⁷⁷http://nlp.stanford.edu/software/corenlp.shtml. is used for lemmatization and tokenization. Identical procedures are applied to all corpora used for probability estimation.

Binomial test is used for significance testing, and with one exception explicitly noted in Section 4.3, all differences presented are statistically highly significant ().

To study the effect of different types of LK in WSD (Section 3.3), for each inventory sense, we choose synonyms (syn), hypernyms (hpr), and hyponyms (hpo) as extended LK in addition to its gloss. The WSD model is evaluated with gloss-only (glo), individual extended LK sources, and the combination of all four sources (all). The results are listed in Table 2 together with existing results (1st and 2nd correspond to the results of the top two unsupervised methods in each dataset).⁸⁸We excluded the results of UNED () in Senseval-2 because, by using sense frequency information that is only obtainable from sense-annotated corpora, it is essentially a supervised system.

By using only glosses, the proposed model already shows statistically significant improvement over the basic Lesk algorithm (92.4% and 140.5% relative improvement in Senseval-2 coarse- and fine-grained tracks, respectively).⁹⁹Comparisons are made against the simplified Lesk algorithm () without usage examples. The comparison is unavailable in SemEval2007 since we have not found existing experiments with this exact configuration. Moreover, comparison between coarse- and fine-grained tracks reveals interesting properties of different LK sources. Previous hypotheses (Section 3.3) are empirically confirmed that WSD performance benefits most from hyponyms and least from hypernyms. Specifically, highly similar, fine-grained sense candidates apparently share more hypernyms in the fine-grained case than in the coarse-grained case; adding to the generality of hypernyms (both semantic and distributional), we postulate that their probability in the NBM is uniformly inflated among many sense candidates, and hence they decrease in distinguishability. Synonyms might help with regard to semantic specification, though their limited quantity also limits their benefits. These patterns on the LK types are consistent in all three experiments.

When including all four LK sources, our model outperforms the state-of-the-art systems with statistical significance in both coarse-grained tasks. For the fine-grained track, it achieves 2nd place after that of , which used a decision list () on manually selected corpora evidence for each inventory sense, and thus is not subject to loss of distinguishability in the glosses as Lesk variants are.

To evaluate model response to probability estimation of different quality (Section 3.4), source corpora are chosen as the majority value of the doc-source attribute of instances in each dataset, namely, the British National Corpus for Senseval-2 (94%) and the Wall Street Journal for SemEval-2007 (86%). The Brown Corpus is shared by both datasets as the baseline corpus. Figure 1 shows the comparison on the SemEval-2007 dataset. Across all experiments, higher WSD accuracy is consistently witnessed using the source corpus; differences are statistically highly significant except for hpo (which is significant with ).