Learning Word Sense Distributions, Detecting Unattested Senses and Identifying Novel Senses Using Topic Models

The automatic determination of word sense information has been a long-term pursuit of the NLP community []. Word sense distributions tend to be Zipfian, and as such, a simple but surprisingly high-accuracy back-off heuristic for word sense disambiguation (wsd) is to tag each instance of a given word with its predominant sense []. Such an approach requires knowledge of predominant senses; however, word sense distributions — and predominant senses too — vary from corpus to corpus. Therefore, methods for automatically learning predominant senses and sense distributions for specific corpora are required [12].

In this paper, we propose a method which uses topic models to estimate word sense distributions. This method is in principle applicable to all parts of speech, and moreover does not require a parser, a hierarchical sense representation or parallel text. Topic models have been used for wsdin a number of studies [1, 13, 19, 3, 11], but our work extends significantly on this earlier work in focusing on the acquisition of prior word sense distributions (and predominant senses).

Because of domain differences and the skewed nature of word sense distributions, it is often the case that some senses in a sense inventory will not be attested in a given corpus. A system capable of automatically finding such senses could reduce ambiguity, particularly in domain adaptation settings, while retaining rare but nevertheless viable senses. We further propose a method for applying our sense distribution acquisition system to the task of finding unattested senses — i.e., senses that are in the sense inventory but not attested in a given corpus. In contrast to the previous work of McCarthy et al. (2004) on this topic which uses the sense ranking score from to remove low-frequency senses from \smallerWordNet, we focus on finding senses that are unattested in the corpus on the premise that, given accurate disambiguation, rare senses in a corpus contribute to correct interpretation.

Corpus instances of a word can also correspond to senses that are not present in a given sense inventory. This can be due to, for example, words taking on new meanings over time (e.g. the relatively recent senses of tablet and swipe related to touchscreen computers) or domain-specific terms not being included in a more general-purpose sense inventory. A system for automatically identifying such novel senses — i.e. senses that are attested in the corpus but not in the sense inventory — would be a very valuable lexicographical tool for keeping sense inventories up-to-date []. We further propose an application of our proposed method to the identification of such novel senses. In contrast to , the use of topic models makes this possible, using topics as a proxy for sense []. Earlier work on identifying novel senses focused on individual tokens [7], whereas our approach goes further in identifying groups of tokens exhibiting the same novel sense.

There has been a considerable amount of research on representing word senses and disambiguating usages of words in context (wsd) as, in order to produce computational systems that understand and produce natural language, it is essential to have a means of representing and disambiguating word sense. wsdalgorithms require word sense information to disambiguate token instances of a given ambiguous word, e.g. in the form of sense definitions [], semantic relationships [17] or annotated data [20]. One extremely useful piece of information is the word sense prior or expected word sense frequency distribution. This is important because word sense distributions are typically skewed [], and systems do far better when they take bias into account [].

Typically, word frequency distributions are estimated with respect to a sense-tagged corpus such as SemCor [15], a 220,000 word corpus tagged with \smallerWordNet [] senses. Due to the expense of hand tagging, and sense distributions being sensitive to domain and genre, there has been some work on trying to estimate sense frequency information automatically [5, 16, 6]. Much of this work has been focused on ranking word senses to find the predominant sense in a given corpus [16], which is a very powerful heuristic approach to wsd. Most wsdsystems rely upon this heuristic for back-off in the absence of strong contextual evidence []. proposed a method which relies on distributionally similar words (nearest neighbours) associated with the target word in an automatically acquired thesaurus []. The distributional similarity scores of the nearest neighbours are associated with the respective target word senses using a \smallerWordNetsimilarity measure, such as those proposed by  and . The word senses are ranked based on these similarity scores, and the most frequent sense is selected for the corpus that the distributional similarity thesaurus was trained over.

As well as sense ranking for predominant sense acquisition, automatic estimates of sense frequency distribution can be very useful for wsdfor training data sampling purposes [], entropy estimation [], and prior probability estimates, all of which can be integrated within a wsdsystem [5, 6, 12]. Various approaches have been adopted, such as normalizing sense ranking scores to obtain a probability distribution [], using subcategorisation information as an indication of verb sense [12] or alternatively using parallel text [5, 6].

The work of Boyd-Graber and Blei (2007) is highly related in that it extends the method of to provide a generative model which assumes the words in a given document are generated according to the topic distribution appropriate for that document. They then predict the most likely sense for each word in the document based on the topic distribution and the words in context (“corroborators”), each of which, in turn, depends on the document’s topic distribution. Using this approach, they get comparable results to McCarthy et al. when context is ignored (i.e. using a model with one topic), and at most a 1% improvement on SemCor when they use more topics in order to take context into account. Since the results do not improve on McCarthy et al. as regards sense distribution acquisition irrespective of context, we will compare our model with that proposed by McCarthy et al.

Recent work on finding novel senses has tended to focus on comparing diachronic corpora [] and has also considered topic models []. In a similar vein, Peirsman et al. (2010) considered the identification of words having a sense particular to one language variety with respect to another (specifically Belgian and Netherlandic Dutch). In contrast to these studies, we propose a model for comparing a corpus with a sense inventory. Carpuat et al. (2013) exploit parallel corpora to identify words in domain-specific monolingual corpora with previously-unseen translations; the method we propose does not require parallel data.

Our methodology is based on the WSI system described in ,¹¹Based on the implementation available at: https://github.com/jhlau/hdp-wsi which has been shown [] to achieve state-of-the-art results over the WSI tasks from SemEval-2007 [], SemEval-2010 [] and SemEval-2013 []. The system is built around a Hierarchical Dirichlet Process (HDP: ), a non-parametric variant of a Latent Dirichlet Allocation topic model [] where the model automatically optimises the number of topics in a fully-unsupervised fashion over the training data.

To learn the senses of a target lemma, we train a single topic model per target lemma. The system reads in a collection of usages of that lemma, and automatically induces topics (= senses) in the form of a multinomial distribution over words, and per-usage topic assignments (= probabilistic sense assignments) in the form of a multinomial distribution over topics. Following , we assign one topic to each usage by selecting the topic that has the highest cumulative probability density, based on the topic allocations of all words in the context window for that usage.²²This includes all words in the usage sentence except stopwords, which were filtered in the preprocessing step. Note that in their original work, experimented with the use of features extracted from a dependency parser. Due to the computational overhead associated with these features, and the fact that the empirical impact of the features was found to be marginal, we make no use of parser-based features in this paper.³³For hyper-parameters $\alpha$ and $\gamma$, we used 0.1 for both. We did not tune the parameters, and opted to use the default parameters introduced in .

The induced topics take the form of word multinomials, and are often represented by the top-$N$ words in descending order of conditional probability. We interpret each topic as a sense of the target lemma.⁴⁴To avoid confusion, we will refer to the HDP-induced topics as topics, and reserve the term sense to denote senses in a sense inventory. To illustrate this, we give the example of topics induced by the HDP model for network in Table 1.

We refer to this method as HDP-WSIhenceforth.⁵⁵The code used to learn predominant sense and run all experiments described in this paper is available at: https://github.com/jhlau/predom_sense.

In predominant sense acquisition, the task is to learn, for each target lemma, the most frequently occurring word sense in a particular domain or corpus, relative to a predefined sense inventory. The WSI system provides us with a topic allocation per usage of a given word, from which we can derive a distribution of topics over usages and a predominant topic. In order to map this onto the predominant sense, we need to have some way of aligning a topic with a sense. We design our topic–sense alignment methodology with portability in mind — it should be applicable to any sense inventory. As such, our alignment methodology assumes only that we have access to a conventional sense gloss or definition for each sense, and does not rely on ontological/structural knowledge (e.g. the \smallerWordNethierarchy).

To compute the similarity between a sense and a topic, we first convert the words in the gloss/definition into a multinomial distribution over words, based on simple maximum likelihood estimation.⁶⁶Words are tokenised using OpenNLP and lemmatised with Morpha []. We additionally remove the target lemma, stopwords and words that are less than 3 characters in length. We then calculate the Jensen–Shannon divergence between the multinomial distribution (over words) of the gloss and that of the topic, and convert the divergence value into a similarity score by subtracting it from $1$. Formally, the similarity sense $s_i$ and topic $t_j$ is:

where $S$ and $T$ are the multinomial distributions over words for sense $s_i$ and topic $t_j$, respectively, and $\text{JS}(X\parallel Y)$ is the Jensen–Shannon divergence for distribution $X$ and $Y$.

To learn the predominant sense, we compute the prevalence score of each sense and take the sense with the highest prevalence score as the predominant sense. The prevalence score for a sense is computed by summing the product of its similarity scores with each topic (i.e. $\text{sim}\left(s_i,t_j\right)$) and the prior probability of the topic in question (based on maximum likelihood estimation). Formally, the prevalence score of sense $s_i$ is given as follows:

where $f\left(t_j\right)$ is the frequency of topic $t_j$ (i.e. the number of usages assigned to topic $t_j$), and $T$ is the number of topics.

The intuition behind the approach is that the predominant sense should be the sense that has relatively high similarity (in terms of lexical overlap) with high-probability topic(s).

We first test the proposed method over the tasks of predominant sense learning and sense distribution induction, using the \smallerWordNet-tagged dataset of , which is made up of 3 collections of documents: a domain-neutral corpus (BNC), and two domain-specific corpora (SPORTSand FINANCE). For each domain, annotators were asked to sense-annotate a random selection of sentences for each of 40 target nouns, based on \smallerWordNetv1.7. The predominant sense and distribution across senses for each target lemma was obtained by aggregating over the sense annotations. The authors evaluated their method in terms of wsdaccuracy over a given corpus, based on assigning all instances of a target word with the predominant sense learned from that corpus. For the remainder of the paper, we denote their system as MKWC.

To compare our system (HDP-WSI) with MKWC, we apply it to the three datasets of . For each dataset, we use HDP to induce topics for each target lemma, compute the similarity between the topics and the \smallerWordNetsenses (Equation (1)), and rank the senses based on the prevalence scores (Equation (2)). In addition to the wsdaccuracy based on the predominant sense inferred from a particular corpus, we additionally compute: (1) ${\text{Acc}}_{\mathrm{\text{ub}}}$, the upper bound for the first sense-based wsdaccuracy (using the gold standard predominant sense for disambiguation);⁷⁷The upper bound for a wsdapproach which tags all token occurrences of a given word with the same sense, as a first step towards context-sensitive unsupervised wsd. and (2) ERR, the error rate reduction between the accuracy for a given system (${\text{Acc}}_{}$) and the upper bound (${\text{Acc}}_{\mathrm{\text{ub}}}$), calculated as follows:

Looking at the results in Table 2, we see little difference in the results for the two methods, with MKWCperforming better over two of the datasets (BNCand SPORTS) and HDP-WSIperforming better over the third (FINANCE), but all differences are small. Based on the McNemar’s Test with Yates correction for continuity, MKWCis significantly better over BNCand HDP-WSIis significantly better over FINANCE( in both cases), but the difference over SPORTSis not statistically significance ($p>0.1$). Note that there is still much room for improvement with both systems, as we see in the gap between the upper bound (based on perfect determination of the first sense) and the respective system accuracies.

Given that both systems compute a continuous-valued prevalence score for each sense of a target lemma, a distribution of senses can be obtained by normalising the prevalence scores across all senses. The predominant sense learning task of evaluates the ability of a method to identify only the head of this distribution, but it is also important to evaluate the full sense distribution []. To this end, we introduce a second evaluation metric: the Jensen–Shannon (JS) divergence between the inferred sense distribution and the gold-standard sense distribution, noting that smaller values are better in this case, and that it is now theoretically possible to obtain a JS divergence of 0 in the case of a perfect estimate of the sense distribution. Results are presented in Table 3.

HDP-WSIconsistently achieves lower JS divergence, indicating that the distribution of senses that it finds is closer to the gold standard distribution. Testing for statistical significance over the paired JS divergence values for each lemma using the Wilcoxon signed-rank test, the result for FINANCEis significant () but the results for the other two datasets are not ($p>0.1$ in each case).

To summarise, the results for MKWCand HDP-WSIare fairly even for predominant sense learning (each outperforms the other at a level of statistical significance over one dataset), but HDP-WSIis better at inducing the overall sense distribution.

It is important to bear in mind that MKWCin these experiments makes use of full-text parsing in calculating the distributional similarity thesaurus, and the \smallerWordNetgraph structure in calculating the similarity between associated words and different senses. Our method, on the other hand, uses no parsing, and only the synset definitions (and not the graph structure) of \smallerWordNet.⁸⁸ obtained good results with definition overlap, but their implementation uses the relation structure alongside the definitions []. Iida et al. (2008) demonstrate that further extensions using distributional data are required when applying the method to resources without hierarchical relations. The non-reliance on parsing is significant in terms of portability to text sources which are less amenable to parsing (such as Twitter: []), and the non-reliance on the graph structure of \smallerWordNetis significant in terms of portability to conventional “flat” sense inventories. While comparable results on a different dataset have been achieved with a proximity thesaurus [] compared to a dependency one,⁹⁹The thesauri used in the reimplementation of MKWCin this paper were obtained from http://webdocs.cs.ualberta.ca/~lindek/downloads.htm. it is not stated how wide a window is needed for the proximity thesaurus. This could be a significant issue with Twitter data, where context tends to be limited. In the next section, we demonstrate the robustness of the method in experimenting with two new datasets, based on Twitter and a web corpus, and the \smallerMacmillan English Dictionary.

In our second set of experiments, we move to a new dataset [9] based on text from ukWaC [8] and Twitter, and annotated using the \smallerMacmillan English Dictionary¹⁰¹⁰http://www.macmillandictionary.com/ (henceforth “\smallerMacmillan”). For the purposes of this research, the choice of \smallerMacmillanis significant in that it is a conventional dictionary with sense definitions and examples, but no linking between senses.¹¹¹¹Strictly speaking, there is limited linking in the form of sets of synonyms in \smallerMacmillan, but we choose to not use this information in our research. In terms of the original research which gave rise to the sense-tagged dataset, \smallerMacmillanwas chosen over \smallerWordNetfor reasons including: (1) the well-documented difficulties of sense tagging with fine-grained \smallerWordNetsenses []; (2) the regular update cycle of \smallerMacmillan(meaning it contains many recently-emerged senses); and (3) the finding in a preliminary sense-tagging task that it better captured Twitter usages than \smallerWordNet(and also \smallerOntoNotes: ).

The dataset is made up of 20 target nouns which were selected to span the high- to mid-frequency range in both Twitter and the ukWaC corpus, and have at least 3 \smallerMacmillansenses. The average sense ambiguity of the 20 target nouns in \smallerMacmillanis 5.6 (but 12.3 in \smallerWordNet). 100 usages of each target noun were sampled from each of Twitter (from a crawl over the time period Jan 3–Feb 28, 2013 using the Twitter Streaming API) and ukWaC, after language identification using langid.py[] and POS tagging (based on the CMU ARK Twitter POS tagger v2.0 [] for Twitter, and the POS tags provided with the corpus for ukWaC). Amazon Mechanical Turk (AMT) was then used to 5-way sense-tag each usage relative to \smallerMacmillan, including allowing the annotators the option to label a usage as “Other” in instances where the usage was not captured by any of the \smallerMacmillansenses. After quality control over the annotators/annotations (see Gella et al. (to appear) for details), and aggregation of the annotations into a single sense per usage (possibly “Other”), there were 2000 sense-tagged ukWaC sentences and Twitter messages over the 20 target nouns. We refer to these two datasets as ukWaCand Twitterhenceforth.

To apply our method to the two datasets, we use HDP-WSIto train a model for each target noun, based on the combined set of usages of that lemma in each of the two background corpora, namely the original Twitter crawl that gave rise to the Twitterdataset, and all of ukWaC.

Our methodologies for the two proposed tasks of identifying unused and novel senses are simple extensions to demonstrate the flexibility and robustness of our methodology. Future work could pursue a more sophisticated methodology, using non-linear combinations of $\text{sim}\left(s_i,t_j\right)$ for computing the affinity measures or multiple features in a supervised context. We contend, however, that these extensions are ultimately a preliminary demonstration to the flexibility and robustness of our methodology.

A natural next step for this research would be to couple sense distribution estimation and the detection of unattested senses with evidence from the context, using topics or other information about the local context (e.g. ) to carry out unsupervised wsdof individual token occurrences of a given word.

In summary, we have proposed a topic modelling-based method for estimating word sense distributions, based on Hierarchical Dirichlet Processes and the earlier work of on word sense induction, in probabilistically mapping the automatically-learned topics to senses in a sense inventory. We evaluated the ability of the method to learn predominant senses and induce word sense distributions, based on a broad range of datasets and two separate sense inventories. In doing so, we established that our method is comparable to the approach of at predominant sense learning, and superior at inducing word sense distributions. We further demonstrated the applicability of the method to the novel tasks of detecting word senses which are unattested in a corpus, and identifying novel senses which are found in a corpus but not captured in a word sense inventory.