Dependency-Based Word Embeddings

Word representation is central to natural language processing. The default approach of representing words as discrete and distinct symbols is insufficient for many tasks, and suffers from poor generalization. For example, the symbolic representation of the words “pizza” and “hamburger” are completely unrelated: even if we know that the word “pizza” is a good argument for the verb “eat”, we cannot infer that “hamburger” is also a good argument. We thus seek a representation that captures semantic and syntactic similarities between words. A very common paradigm for acquiring such representations is based on the distributional hypothesis of Harris [16], stating that words in similar contexts have similar meanings.

Based on the distributional hypothesis, many methods of deriving word representations were explored in the NLP community. On one end of the spectrum, words are grouped into clusters based on their contexts [5, 32]. On the other end, words are represented as a very high dimensional but sparse vectors in which each entry is a measure of the association between the word and a particular context (see [30, 3] for a comprehensive survey). In some works, the dimensionality of the sparse word-context vectors is reduced, using techniques such as SVD [6] or LDA [25, 27, 8]. Most recently, it has been proposed to represent words as dense vectors that are derived by various training methods inspired from neural-network language modeling [4, 10, 23, 20, 21]. These representations, referred to as “neural embeddings” or “word embeddings”, have been shown to perform well across a variety of tasks [29, 9, 26, 2].

Word embeddings are easy to work with because they enable efficient computation of word similarities through low-dimensional matrix operations. Among the state-of-the-art word-embedding methods is the skip-gram with negative sampling model (SkipGram), introduced by Mikolov et al. [21] and implemented in the word2vec software.¹¹code.google.com/p/word2vec/ Not only does it produce useful word representations, but it is also very efficient to train, works in an online fashion, and scales well to huge copora (billions of words) as well as very large word and context vocabularies.

Previous work on neural word embeddings take the contexts of a word to be its linear context – words that precede and follow the target word, typically in a window of $k$ tokens to each side. However, other types of contexts can be explored too.

In this work, we generalize the SkipGram model, and move from linear bag-of-words contexts to arbitrary word contexts. Specifically, following work in sparse vector-space models [18, 24, 3], we experiment with syntactic contexts that are derived from automatically produced dependency parse-trees.

The different kinds of contexts produce noticeably different embeddings, and induce different word similarities. In particular, the bag-of-words nature of the contexts in the “original” SkipGram model yield broad topical similarities, while the dependency-based contexts yield more functional similarities of a cohyponym nature. This effect is demonstrated using both qualitative and quantitative analysis (Section 4).

The neural word-embeddings are considered opaque, in the sense that it is hard to assign meanings to the dimensions of the induced representation. In Section 5 we show that the SkipGram model does allow for some introspection by querying it for contexts that are “activated by” a target word. This allows us to peek into the learned representation and explore the contexts that are found by the learning process to be most discriminative of particular words (or groups of words). To the best of our knowledge, this is the first work to suggest such an analysis of discriminatively-trained word-embedding models.

Our departure point is the skip-gram neural embedding model introduced in [19] trained using the negative-sampling procedure presented in [21]. In this section we summarize the model and training objective following the derivation presented by Goldberg and Levy [13], and highlight the ease of incorporating arbitrary contexts in the model.

In the skip-gram model, each word $w\in W$ is associated with a vector $v_w\in R^d$ and similarly each context $c\in C$ is represented as a vector $v_c\in R^d$, where $W$ is the words vocabulary, $C$ is the contexts vocabulary, and $d$ is the embedding dimensionality. The entries in the vectors are latent, and treated as parameters to be learned. Loosely speaking, we seek parameter values (that is, vector representations for both words and contexts) such that the dot product $v_w\cdot v_c$ associated with “good” word-context pairs is maximized.

More specifically, the negative-sampling objective assumes a dataset $D$ of observed $\left(w,c\right)$ pairs of words $w$ and the contexts $c$, which appeared in a large body of text. Consider a word-context pair $\left(w,c\right)$. Did this pair come from the data? We denote by $p(D=1|w,c)$ the probability that $\left(w,c\right)$ came from the data, and by $p(D=0|w,c)=1-p(D=1|w,c)$ the probability that $\left(w,c\right)$ did not. The distribution is modeled as:

where $v_w$ and $v_c$ (each a $d$-dimensional vector) are the model parameters to be learned. We seek to maximize the log-probability of the observed pairs belonging to the data, leading to the objective:

This objective admits a trivial solution in which $p(D=1|w,c)=1$ for every pair $\left(w,c\right)$. This can be easily achieved by setting $v_c=v_w$ and $v_c\cdot v_w=K$ for all $c,w$, where $K$ is large enough number.

In order to prevent the trivial solution, the objective is extended with $\left(w,c\right)$ pairs for which $p(D=1|w,c)$ must be low, i.e. pairs which are not in the data, by generating the set $D^{\prime }$ of random $\left(w,c\right)$ pairs (assuming they are all incorrect), yielding the negative-sampling training objective:

which can be rewritten as:

where $\sigma \left(x\right)=1/\left(1+e^x\right)$. The objective is trained in an online fashion using stochastic-gradient updates over the corpus $D\cup D^{\prime }$.

The negative samples $D^{\prime }$ can be constructed in various ways. We follow the method proposed by Mikolov et al.: for each $\left(w,c\right)\in D$ we construct $n$ samples $\left(w,c_1\right),\mathrm{\dots },\left(w,c_n\right)$, where $n$ is a hyperparameter and each $c_j$ is drawn according to its unigram distribution raised to the $3/4$ power.

Optimizing this objective makes observed word-context pairs have similar embeddings, while scattering unobserved pairs. Intuitively, words that appear in similar contexts should have similar embeddings, though we have not yet found a formal proof that SkipGram does indeed maximize the dot product of similar words.

In the SkipGram embedding algorithm, the contexts of a word $w$ are the words surrounding it in the text. The context vocabulary $C$ is thus identical to the word vocabulary $W$. However, this restriction is not required by the model; contexts need not correspond to words, and the number of context-types can be substantially larger than the number of word-types. We generalize SkipGram by replacing the bag-of-words contexts with arbitrary contexts.

In this paper we experiment with dependency-based syntactic contexts. Syntactic contexts capture different information than bag-of-word contexts, as we demonstrate using the sentence “Australian scientist discovers star with telescope”.

We experiment with 3 training conditions: BoW5 (bag-of-words contexts with $k=5$), BoW2 (same, with $k=2$) and Deps (dependency-based syntactic contexts). We modified word2vec to support arbitrary contexts, and to output the context embeddings in addition to the word embeddings. For bag-of-words contexts we used the original word2vec implementation, and for syntactic contexts, we used our modified version. The negative-sampling parameter (how many negative contexts to sample for every correct one) was 15.

All embeddings were trained on English Wikipedia. For Deps, the corpus was tagged with parts-of-speech using the Stanford tagger [28] and parsed into labeled Stanford dependencies [11] using an implementation of the parser described in [14]. All tokens were converted to lowercase, and words and contexts that appeared less than 100 times were filtered. This resulted in a vocabulary of about 175,000 words, with over 900,000 distinct syntactic contexts. We report results for 300 dimension embeddings, though similar trends were also observed with 600 dimensions.

Neural word embeddings are often considered opaque and uninterpretable, unlike sparse vector space representations in which each dimension corresponds to a particular known context, or LDA models where dimensions correspond to latent topics. While this is true to a large extent, we observe that SkipGram does allow a non-trivial amount of introspection. Although we cannot assign a meaning to any particular dimension, we can indeed get a glimpse at the kind of information being captured by the model, by examining which contexts are “activated” by a target word.

Recall that the learning procedure is attempting to maximize the dot product $v_c\cdot v_w$ for good $\left(w,c\right)$ pairs and minimize it for bad ones. If we keep the context embeddings, we can query the model for the contexts that are most activated by (have the highest dot product with) a given target word. By doing so, we can see what the model learned to be a good discriminative context for the word.

To demonstrate, we list the 5 most activated contexts for our example words with Deps embeddings in Table 2. Interestingly, the most discriminative syntactic contexts in these cases are not associated with subjects or objects of verbs (or their inverse), but rather with conjunctions, appositions, noun-compounds and adjectivial modifiers. Additionally, the collapsed preposition relation is very useful (e.g. for capturing the school aspect of hogwarts). The presence of many conjunction contexts, such as superman/conj for batman and singing/conj for dancing, may explain the functional similarity observed in Section 4; conjunctions in natural language tend to enforce their conjuncts to share the same semantic types and inflections.

In the future, we hope that insights from such model introspection will allow us to develop better contexts, by focusing on conjunctions and prepositions for example, or by trying to figure out why the subject and object relations are absent and finding ways of increasing their contributions.

We presented a generalization of the SkipGram embedding model in which the linear bag-of-words contexts are replaced with arbitrary ones, and experimented with dependency-based contexts, showing that they produce markedly different kinds of similarities. These results are expected, and follow similar findings in the distributional semantics literature. We also demonstrated how the resulting embedding model can be queried for the discriminative contexts for a given word, and observed that the learning procedure seems to favor relatively local syntactic contexts, as well as conjunctions and objects of preposition. We hope these insights will facilitate further research into improved context modeling and better, possibly task-specific, embedded representations. Our software, allowing for experimentation with arbitrary contexts, together with the embeddings described in this paper, are available for download at the authors’ websites.