Improving Lexical Embeddings with Semantic Knowledge

Word2vec [] is an algorithm for learning embeddings using a neural language model. Embeddings are represented by a set of latent (hidden) variables, and each word is represented by a specific instantiation of these variables. Training learns these representations for each word $w_t$ (the $t$th word in a corpus of size $T$) so as to maximize the log likelihood of each token given its context: words within a window sized $c$:

where $w_{t-c}^{t+c}$ is the set of words in the window of size $c$ centered at $w_t$ ($w_t$ excluded).

Word2vec offers two choices for modeling of Eq. (1): a skip-gram model and a continuous bag-of-words model (cbow). The latter worked better in our experiments so we focus on it in our presentation. cbow defines $p\left(w_t\right|w_{t-c}^{t+c})$ as:

where $e_w$ and $e_w^{{}^{\prime }}$ represent the input and output embeddings respectively, i.e., the assignments to the latent variables for word $w$. While some learn a single representation for each word ($e_w^{\prime }\triangleq e_w$), our results improved when we used a separate embedding for input and output in cbow.

Suppose we have a resource that indicates relations between words. In the case of semantics, we could have a resource that encodes semantic similarity between words. Based on this resource, we learn embeddings that predict one word from another related word. We define $𝐑$ as a set of relations between two words $w$ and $w^{\prime }$. $𝐑$ can contain typed relations (e.g., $w$ is related to $w^{\prime }$ through a specific type of semantic relation), and relations can have associated scores indicating their strength. We assume a single relation type of uniform strength, though it is straightforward to include additional characteristics into the objective.

Define ${𝐑}_w$ to be the subset of relations in $𝐑$ which involve word $w$. Our objective maximizes the (log) probability of all relations by summing over all words $N$ in the vocabulary:

$p\left(w\right|w_i)=\exp \left(e_w^{\prime }{}^T{e}_{w_i}\right)/{\sum }_{\overline{w}}\exp \left(e_{\overline{w}}^{\prime }{}^T{e}_{w_i}\right)$ takes a form similar to Eq. (2) but without the context: $e$ and $e^{\prime }$ are again the input and output embeddings. For our semantic relations $e_w^{\prime }$ and $e_w$ are symmetrical, so we use a single embedding. Embeddings are learned such that they are predictive of related words in the resource. We call this the Relation Constrained Model (RCM).

The cbow and RCM objectives use separate data for learning. While RCM learns embeddings suited to specific tasks based on knowledge resources, cbow learns embeddings for words not included in the resource but appear in a corpus. We form a joint model through a linear combination of the two (weighted by $C$):

Based on our initial experiments, RCM uses the output embeddings of cbow.

We learn embeddings using stochastic gradient ascent. Updates for the first term for $e^{\prime }$ and $e$ are:

where $\sigma \left(x\right)=\exp \left\{x\right\}/\left(1+\exp \left\{x\right\}\right)$, $I_{\left[x\right]}$ is 1 when $x$ is true, $f\left(w\right)=e_w^{\prime }{}^{\top }{\sum }_{j=t-c}^{t+c}{e}_{w_j}$. Second term updates are:

where $f^{\prime }\left(w\right)=e_w^{\prime }{}^{\top }{e}_{w_i}^{\prime }$. We use two learning rates: ${\alpha }_{\text{cbow}}$ and ${\alpha }_{\text{RCM}}$.

All three models (cbow, RCM and joint) use the same training scheme based on . There are several choices to make in parameter estimation; we present the best performing choices used in our results.

We use noise contrastive estimation (NCE) [], which approximately maximizes the log probability of the softmax objective (Eq. 2). For each objective (cbow or RCM), we sample 15 words as negative samples for each training instance according to their frequencies in raw texts (i.e. training data of cbow). Suppose $w$ has frequency $u\left(w\right)$, then the probability of sampling $w$ is $p\left(w\right)\propto u{\left(w\right)}^{3/4}$.

We use distributed training, where shared embeddings are updated by each thread based on training data within the thread, i.e., asynchronous stochastic gradient ascent. For the joint model, we assign threads to the cbow or RCM objective with a balance of 12:1(i.e. $C$ is approximately $\frac{1}{12}$). We allow the cbow threads to control convergence; training stops when these threads finish processing the data. We found this an effective method for balancing the two objectives. We trained each cbow objective using a single pass over the data set (except for those in Section 4.1), which we empirically verified was sufficient to ensure stable performances on semantic tasks.

Model pre-training is critical in deep learning []. We evaluate two strategies: random initialization, and pre-training the embeddings. For pre-training, we first learn using cbow with a random initialization. The resulting trained model is then used to initialize the RCM model. This enables the RCM model to benefit from the unlabeled data, but refine the embeddings constrained by the given relations.

Finally, we consider a final model for training embeddings that uses a specific training regime. While the joint model balances between fitting the text and learning relations, modeling the text at the expense of the relations may negatively impact the final embeddings for tasks that use the embeddings outside of the context of word2vec. Therefore, we use the embeddings from a trained joint model to pre-train an RCM model. We call this setting Joint$\to$RCM.

Word2vec is fundamentally a language model, which allows us to compute standard evaluation metrics on a held out dataset. After obtaining trained embeddings from any of our objectives, we use the embeddings in the word2vec model to measure perplexity of the test set. Measuring perplexity means computing the exact probability of each word, which requires summation over all words in the vocabulary in the denominator of the softmax. Therefore, we also trained the language models with hierarchical classification [] strategy (HS). The averaged perplexities are reported on the NYT test set.

While word2vec and joint are trained as language models, RCM is not. In fact, RCM does not even observe all the words that appear in the training set, so it makes little sense to use the RCM embeddings directly for language modeling. Therefore, in order to make fair comparison, for every set of trained embeddings, we fix them as input embedding for word2vec, then learn the remaining input embeddings (words not in the relations) and all the output embeddings using cbow. Since this involves running cbow on NYT data for 2 iterations (one iteration for word2vec-training/pre-training/joint-modeling and the other for tuning the language model), we use Joint-r (random initialization) for a fair comparison.

Table 2 shows the results for language modeling on test data. All of our proposed models improve over the baseline in terms of perplexity when NCE is used for training LMs. When HS is used, the perplexities are greatly improved. However in this situation only the joint models improve the results; and Joint$\to$RCM performs similar to the baseline, although it is not designed for language modeling. We include the optimal ${\alpha }_{\text{RCM}}$ in the table; while set ${\alpha }_{\text{cbow}}=0.025$ (the default setting of word2vec). Even when our goal is to strictly model the raw text corpus, we obtain improvements by injecting semantic information into the objective. RCM can effectively shift learning to obtain more informative embeddings.

Our next task is to find semantically related words using the embeddings, evaluating on relations from PPDB and WordNet. For each of the word pairs in the evaluation set A,B$>$, we use the cosine distance between the embeddings to score $A$ with a candidate word $B^{\prime }$. We use a large sample of candidate words (10k, 30k or 100k) and rank all candidate words for pairs where $B$ appears in the candidates. We then measure the rank of the correct $B$ to compute mean reciprocal rank (MRR). Our goal is to use word $A$ to select word $B$ as the closest matching word from the large set of candidates. Using this strategy, we evaluate the embeddings from all of our objectives and measure which embedding most accurately selected the true correct word.

Table 3 shows MRR results for both PPDB and WordNet dev and test datasets for all models. All of our methods improve over the baselines in nearly every test set result. In nearly every case, Joint$\to$RCM obtained the largest improvements. Clearly, our embeddings are much more effective at capturing semantic similarity.

Our final evaluation is to predict human judgements of semantic relatedness. We have pairs of words from PPDB scored by annotators on a scale of 1 to 5 for quality of similarity. Our data are the judgements used by , which we filtered to include only those pairs for which we learned embeddings, yielding 868 pairs.

We assign a score using the dot product between the output embeddings of each word in the pair, then order all 868 pairs according to this score. Using the human judgements, we compute the swapped pairs rate: the ratio between the number of swapped pairs and the number of all pairs. For pair $p$ scored $y_p$ by the embeddings and judged ${\widehat{y}}_p$ by an annotator, the swapped pair rate is:

where $\text{I}\left[x\right]$ is 1 when $x$ is true.

Table 4 shows that all of our models obtain reductions in error as compared to the baseline (cbow), with Joint$\to$RCM obtaining the largest reduction. This suggests that our embeddings are better suited for semantic tasks, in this case judged by human annotations.

We conclude our experiments with an analysis of modeling choices. First, pre-training RCM models gives significant improvements in both measuring semantic similarity and capturing human judgements (compare “p” vs. “r” results.) Second, the number of relations used for RCM training is an important factor. Table 5 shows the effect on dev data of using various numbers of relations. While we see improvements from XL to XXL (5 times as many relations), we get worse results on XXXL, likely because this set contains the lowest quality relations in PPDB. Finally, Table 6 shows different learning rates ${\alpha }_{\text{RCM}}$ for the RCM objective.

The baseline word2vec and the joint model have nearly the same averaged running times (2,577s and 2,644s respectively), since they have same number of threads for the CBOW objective and the joint model uses additional threads for the RCM objective. The RCM models are trained with single thread for 100 epochs. When trained on the PPDB-XXL data, it spends 2,931s on average.