Learning to Predict Distributions of Words Across Domains

Before we tackle the problem of learning a model to predict the distribution of a word across domains, we must first compute the distribution of a word from a single domain. For this purpose, we represent a word $w$ using unigrams and bigrams that co-occur with $w$ in a sentence as follows.

Given a document H, such as a user-review of a product, we split H into sentences, and lemmatize each word in a sentence using the RASP system []. Using a standard stop word list, we filter out frequent non-content unigrams and select the remainder as unigram features to represent a sentence. Next, we generate bigrams of word lemmas and remove any bigrams that consists only of stop words. Bigram features capture negations more accurately than unigrams, and have been found to be useful for sentiment classification tasks. Table 1 shows the unigram and bigram features we extract for a sentence using this procedure. Using data from a single domain, we construct a feature co-occurrence matrix $\text{mat}A$ in which columns correspond to unigram features and rows correspond to either unigram or bigram features. The value of the element $a_{ij}$ in the co-occurrence matrix $\text{mat}A$ is set to the number of sentences in which the $i$-th and $j$-th features co-occur.

Typically, the number of unique bigrams is much larger than that of unigrams. Moreover, co-occurrences of bigrams are rare compared to co-occurrences of unigrams, and co-occurrences involving a unigram and a bigram. Consequently, in matrix $\text{mat}A$, we consider co-occurrences only between unigrams vs. unigrams, and bigrams vs. unigrams. We consider each row in $\text{mat}A$ as representing the distribution of a feature (i.e. unigrams or bigrams) in a particular domain over the unigram features extracted from that domain (represented by the columns of $\text{mat}A$). We apply Positive Pointwise Mutual Information (PPMI) to the co-occurrence matrix $\text{mat}A$. This is a variation of the Pointwise Mutual Information (PMI) [], in which all PMI values that are less than zero are replaced with zero []. Let $\text{mat}F$ be the matrix that results when PPMI is applied to $\text{mat}A$. Matrix $\text{mat}F$ has the same number of rows, $n_r$, and columns, $n_c$, as the raw co-occurrence matrix $\text{mat}A$.

Note that in addition to the above-mentioned representation, there are many other ways to represent the distribution of a word in a particular domain []. For example, one can limit the definition of co-occurrence to words that are linked by some dependency relation [], or extend the window of co-occurrence to the entire document []. Since the method we propose in Section 3.2 to predict the distribution of a word across domains does not depend on the particular feature representation method, any of these alternative methods could be used.

To reduce the dimensionality of the feature space, and create dense representations for words, we perform SVD on $\text{mat}F$. We use the left singular vectors corresponding to the $k$ largest singular values to compute a rank $k$ approximation $\widehat{\text{mat}F}$, of $\text{mat}F$. We perform truncated SVD using SVDLIBC²²http://tedlab.mit.edu/~dr/SVDLIBC/. Each row in $\widehat{\text{mat}F}$ is considered as representing a word in a lower $k\left(\ll n_c\right)$ dimensional feature space corresponding to a particular domain. Distribution prediction in this lower dimensional feature space is preferrable to prediction over the original feature space because there are reductions in overfitting, feature sparseness, and the learning time. We created two matrices, ${\widehat{\text{mat}F}}_{\text{cS}}$ and ${\widehat{\text{mat}F}}_{\text{cT}}$ from the source and target domains, respectively, using the above mentioned procedure.

We propose a method to learn a model that can predict the distribution ${\overrightarrow{w}}_{\text{cT}}$ of a word $w$ in the target domain $\text{cT}$, given its distribution ${\overrightarrow{w}}_{\text{cS}}$ in the source domain $\text{cS}$. We denote the set of features that occur in both domains by $\text{cW}=\left\{w^{\left(1\right)},\mathrm{\dots },w^{\left(n\right)}\right\}$. In the literature, such features are often referred to as pivots, and they have been shown to be useful for DA, allowing the weights learnt to be transferred from one domain to another. Various criteria have been proposed for selecting a small set of pivots for DA, such as the mutual information of a word with the two domains []. However, we do not impose any further restrictions on the set of pivots $\text{cW}$ other than that they occur in both domains.

For each word $w^{\left(i\right)}\in \text{cW}$, we denote the corresponding rows in ${\widehat{\text{mat}F}}_{\text{cS}}$ and ${\widehat{\text{mat}F}}_{\text{cT}}$ by column vectors ${\overrightarrow{w}}_{\text{cS}}^{\left(i\right)}$ and ${\overrightarrow{w}}_{\text{cT}}^{\left(i\right)}$. Note that the dimensionality of ${\overrightarrow{w}}_{\text{cS}}^{\left(i\right)}$ and ${\overrightarrow{w}}_{\text{cT}}^{\left(i\right)}$ need not be equal, and we may select different numbers of singular vectors to approximate ${\widehat{\text{mat}F}}_{\text{cS}}$ and ${\widehat{\text{mat}F}}_{\text{cT}}$. We model distribution prediction as a multivariate regression problem where, given a set ${\left\{\left({\overrightarrow{w}}_{\text{cS}}^{\left(i\right)},{\overrightarrow{w}}_{\text{cT}}^{\left(i\right)}\right)\right\}}_{i=1}^n$ consisting of pairs of feature vectors selected from each domain for the pivots in $\text{cW}$, we learn a mapping from the inputs (${\overrightarrow{w}}_{\text{cS}}^{\left(i\right)}$) to the outputs (${\overrightarrow{w}}_{\text{cT}}^{\left(i\right)}$).

We use Partial Least Squares Regression (PLSR) [] to learn a regression model using pairs of vectors. PLSR has been applied in Chemometrics [], producing stable prediction models even when the number of samples is considerably smaller than the dimensionality of the feature space. In particular, PLSR fits a smaller number of latent variables ($10-100$ in practice) such that the correlation between the feature vectors for pivots in the two domains are maximised in this latent space.

Let $\text{mat}X$ and $\text{mat}Y$ denote matrices formed by arranging respectively the vectors ${\overrightarrow{w}}_{\text{cS}}^{\left(i\right)}$s and ${\overrightarrow{w}}_{\text{cT}}^{\left(i\right)}$ in rows. PLSR decomposes $\text{mat}X$ and $\text{mat}Y$ into a series of products between rank $1$ matrices as follows:

Here, ${\overrightarrow{\lambda }}_l$, ${\overrightarrow{\gamma }}_l$, ${\overrightarrow{p}}_l$, and ${\overrightarrow{q}}_l$ are column vectors, and the summation is taken over the rank $1$ matrices that result from the outer product of those vectors. The matrices, $𝚲$, $𝚪$, $\text{mat}P$, and $\text{mat}Q$ are constructed respectively by arranging ${\overrightarrow{\lambda }}_l$, ${\overrightarrow{\gamma }}_l$, ${\overrightarrow{p}}_l$, and ${\overrightarrow{q}}_l$ vectors as columns.

[t] Learning a prediction model. {algorithmic}[1] \REQUIRE$\text{mat}X$, $\text{mat}Y$, $L$. \ENSUREPrediction matrix $\text{mat}M$. \STATERandomly select ${\overrightarrow{\gamma }}_l$ from columns in $\text{mat}{Y}_l$. \STATE${\overrightarrow{v}}_l=\text{mat}{X}_l\text{T}{\overrightarrow{\gamma }}_l/\text{norm}\text{mat}{X}_l\text{T}{\overrightarrow{\gamma }}_l$ \STATE${\overrightarrow{\lambda }}_l=\text{mat}{X}_l{\overrightarrow{v}}_l$ \STATE${\overrightarrow{q}}_l=\text{mat}{Y}_l\text{T}{\overrightarrow{\lambda }}_l/\text{norm}\text{mat}{Y}_l\text{T}{\overrightarrow{\lambda }}_l$ \STATE${\overrightarrow{\gamma }}_l=\text{mat}{Y}_l{\overrightarrow{q}}_l$ \STATEIf ${\overrightarrow{\gamma }}_l$ is unchanged go to Line 3.2; otherwise go to Line 3.2 \STATE$c_l={\overrightarrow{\lambda }}_l\text{T}{\overrightarrow{\gamma }}_l/\text{norm}{\overrightarrow{\lambda }}_l\text{T}{\overrightarrow{\gamma }}_l$ \STATE${\overrightarrow{p}}_l=\text{mat}{X}_l\text{T}{\overrightarrow{\lambda }}_l/{\overrightarrow{\lambda }}_l\text{T}{\overrightarrow{\lambda }}_l$ \STATE$\text{mat}{X}_{l+1}=\text{mat}{X}_l-{\overrightarrow{\lambda }}_l{\overrightarrow{p}}_l\text{T}$ and $\text{mat}{Y}_{l+1}=\text{mat}{Y}_l-c_l{\overrightarrow{\lambda }}_l{\overrightarrow{q}}_l\text{T}$. \STATEStop if $l=L$; otherwise $l=l+1$ and return to Line 3.2. \STATELet $\text{mat}C=\mathrm{diag}(c_1,\mathrm{\dots },c_L)$, and $\text{mat}V=[{\overrightarrow{v}}_1\mathrm{\dots }{\overrightarrow{v}}_L]$ \STATE$\text{mat}M=\text{mat}V(\text{mat}P\text{T}\text{mat}V)\text{inv}\text{mat}C\text{mat}Q\text{T}$ \RETURN$\text{mat}M$ Our method for learning a distribution prediction model is shown in Algorithm 3.2. It is based on the two block NIPALS routine [] and iteratively discovers $L$ pairs of vectors (${\overrightarrow{\lambda }}_l,{\overrightarrow{\gamma }}_l)$ such that the covariances, $\mathrm{Cov}\left({\overrightarrow{\lambda }}_l,{\overrightarrow{\gamma }}_l\right)$, are maximised under the constraint $\text{norm}{\overrightarrow{p}}_l=\text{norm}{\overrightarrow{q}}_l=1$. Finally, the prediction matrix, $\text{mat}M$ is computed using ${\overrightarrow{\lambda }}_l,{\overrightarrow{\gamma }}_l,{\overrightarrow{p}}_l,{\overrightarrow{q}}_l$. The predicted distribution ${\widehat{\overrightarrow{w}}}_{\text{cT}}$ of a word $w$ in $\text{cT}$ is given by

Our distribution prediction learning method is unsupervised in the sense that it does not require manually labeled data for a particular task from any of the domains. This is an important point, and means that the distribution prediction method is independent of the task to which it may subsequently be applied. As we go on to show in Section 6, this enables us to use the same distribution prediction method for both POS tagging and sentiment classification.

We represent each word using a set of features such as capitalisation (whether the first letter of the word is capitalised), numeric (whether the word contains digits), prefixes up to four letters, and suffixes up to four letters []. Next, for each word $w$ in a source domain labeled (i.e. manually POS tagged) sentence, we select its neighbours $u^{\left(i\right)}$ in the source domain as additional features. Specifically, we measure the similarity, $\mathrm{sim}\left({\overrightarrow{u}}_{\text{cS}}^{\left(i\right)},{\overrightarrow{w}}_{\text{cS}}\right)$, between the source domain distributions of $u^{\left(i\right)}$ and $w$, and select the top $r$ similar neighbours $u^{\left(i\right)}$ for each word $w$ as additional features for $w$. We refer to such features as distributional features in this work. The value of a neighbour $u^{\left(i\right)}$ selected as a distributional feature is set to its similarity score $\mathrm{sim}\left({\overrightarrow{u}}_{\text{cS}}^{\left(i\right)},{\overrightarrow{w}}_{\text{cS}}\right)$. Next, we train a CRF model using all features (i.e. capitalisation, numeric, prefixes, suffixes, and distributional features) on source domain labeled sentences.

We train a PLSR model, $\text{mat}M$, that predicts the target domain distribution $\text{mat}M{\overrightarrow{u}}_{\text{cS}}^{\left(i\right)}$ of a word $u^{\left(i\right)}$ in the source domain labeled sentences, given its distribution, ${\overrightarrow{u}}_{\text{cS}}^{\left(i\right)}$. At test time, for each word $w$ that appears in a target domain test sentence, we measure the similarity, $\mathrm{sim}\left(\text{mat}M{\overrightarrow{u}}_{\text{cS}}^{\left(i\right)},{\overrightarrow{w}}_{\text{cT}}\right)$, and select the most similar $r$ words $u^{\left(i\right)}$ in the source domain labeled sentences as the distributional features for $w$, with their values set to $\mathrm{sim}\left(\text{mat}M{\overrightarrow{u}}_{\text{cS}}^{\left(i\right)},{\overrightarrow{w}}_{\text{cT}}\right)$. Finally, the trained CRF model is applied to a target domain test sentence.

Note that distributional features are always selected from the source domain during both train and test times, thereby increasing the number of overlapping features between the trained model and test sentences. To make the inference tractable and efficient, we use a first-order Markov factorisation, in which we consider all pairwise combinations between the features for the current word and its immediate predecessor.

Unlike in POS tagging, where we must individually tag each word in a target domain test sentence, in sentiment classification we must classify the sentiment for the entire review. We modify the DA method presented in Section 4.1 to satisfy this requirement as follows.

Let us assume that we are given a set ${\left\{\left({\overrightarrow{x}}_{\text{cS}}^{\left(i\right)},y^{\left(i\right)}\right)\right\}}_{i=1}^n$ of $n$ labeled reviews ${\overrightarrow{x}}_{\text{cS}}^{\left(i\right)}$ for the source domain $\text{cS}$. For simplicity, let us consider binary sentiment classification where each review ${\overrightarrow{x}}^{\left(i\right)}$ is labeled either as positive (i.e. $y^{\left(i\right)}=1$) or negative (i.e. $y^{\left(i\right)}=-1$). Our cross-domain binary sentiment classification method can be easily extended to the multi-class setting as well. First, we lemmatise each word in a source domain labeled review ${\overrightarrow{x}}_{\text{cS}}^{\left(i\right)}$, and extract both unigrams and bigrams as features to represent ${\overrightarrow{x}}_{\text{cS}}^{\left(i\right)}$ by a binary-valued feature vector. Next, we train a binary classification model, $\overrightarrow{\theta }$, using those feature vectors. Any binary classification algorithm can be used to learn $\overrightarrow{\theta }$. In our experiments, we used L2 regularised logistic regression.

Next, we train a PLSR model, $\text{mat}M$, as described in Section 3.2 using unlabeled reviews in the source and target domains. At test time, we represent a test target review H using a binary-valued feature vector $\overrightarrow{h}$ of unigrams and bigrams of lemmas of the words in H, as we did for source domain labeled train reviews. Next, for each feature $w^{\left(j\right)}$ extracted from H, we measure the similarity, $\mathrm{sim}\left(\text{mat}M{\overrightarrow{u}}_{\text{cS}}^{\left(i\right)},{\overrightarrow{w}}_{\text{cT}}^{\left(j\right)}\right)$, between the target domain distribution of $w^{\left(j\right)}$, and each feature (unigram or bigram) $u^{\left(i\right)}$ in the source domain labeled reviews. We score each source domain feature $u^{\left(i\right)}$ for its relatedness to H using the formula:

where $\left|\text{𝖧}\right|$ denotes the total number of features extracted from the test review H. We select the top scoring $r$ features $u^{\left(i\right)}$ as distributional features for H, and append those to $\overrightarrow{h}$. The corresponding values of those distributional features are set to the scores given by Equation 4. Finally, we classify $\overrightarrow{h}$ using the trained binary classifier $\overrightarrow{\theta }$. Note that given a test review, we find the distributional features that are similar to all the words in the test review from the source domain. In particular, we do not find distributional features independently for each word in the test review. This enables us to find distributional features that are consistent with all the features in a test review.

For both POS tagging and sentiment classification, we experimented with several alternative approaches for feature weighting, representation, and similarity measures using development data, which we randomly selected from the training instances from the datasets described in Section 5.

For feature weighting for sentiment classification, we considered using the number of occurrences of a feature in a review and tf-idf weighting []. For representation, we considered distributional features $u^{\left(i\right)}$ in descending order of their scores given by Equation 4, and then taking the inverse-rank as the values for the distributional features []. However, none of these alternatives resulted in performance gains. With respect to similarity measures, we experimented with cosine similarity and the similarity measure proposed by ; cosine similarity performed consistently well over all the experimental settings. The feature representation was held fixed during these similarity measure comparisons.

For POS tagging, we measured the effect of varying $r$, the number of distributional features, using a development dataset. We observed that setting $r$ larger than $10$ did not result in significant improvements in tagging accuracy, but only increased the train time due to the larger feature space. Consequently, we set $r=10$ in POS tagging. For sentiment analysis, we used all features in the source domain labeled reviews as distributional features, weighted by their scores given by Equation 4, taking the inverse-rank. In both tasks, we parallelised similarity computations using BLAS³³http://www.openblas.net/ level-3 routines to speed up the computations. The source code of our implementation is publicly available⁴⁴http://www.csc.liv.ac.uk/~danushka/software.html.

Table 2 shows the token-level POS tagging accuracy for unseen words (i.e. words that appear in the target domain test sentences but not in the source domain labeled train sentences). By limiting the evaluation to unseen words instead of all words, we can evaluate the gain in POS tagging accuracy solely due to DA. The NA (no-adapt) baseline simulates the effect of not performing any DA. Specifically, in POS tagging, a CRF trained on source domain labeled sentences is applied to target domain test sentences, whereas in sentiment classification, a logistic regression classifier trained using source domain labeled reviews is applied to the target domain test reviews. The ${\text{cS}}_{\mathrm{𝐩𝐫𝐞𝐝}}$ baseline directly uses the source domain distributions for the words instead of projecting them to the target domain. This is equivalent to setting the prediction matrix $\text{mat}M$ to the unit matrix. The ${\text{cT}}_{\mathrm{𝐩𝐫𝐞𝐝}}$ baseline uses the target domain distribution ${\overrightarrow{w}}_{\text{cT}}$ for a word $w$ instead of $\text{mat}M{\overrightarrow{w}}_{\text{cS}}$. If $w$ does not appear in the target domain, then ${\overrightarrow{w}}_{\text{cT}}$ is set to the zero vector. The ${\text{cS}}_{pred}$ and ${\text{cT}}_{pred}$ baselines simulate the two alternatives of using source and target domain distributions instead of learning a PLSR model. The DA method proposed in Section 4.1 is shown as the Proposed method. Filter denotes the training set filtering method proposed by for the DA of POS taggers.

From Table 2, we see that the Proposed method achieves the best performance in all five domains, followed by the ${\text{cT}}_{pred}$ baseline. Recall that the ${\text{cT}}_{pred}$ baseline cannot find source domain words that do not appear in the target domain as distributional features for the words in the target domain test reviews. Therefore, when the overlap between the vocabularies used in the source and the target domains is small, ${\text{cT}}_{pred}$ cannot reduce the mismatch between the feature spaces. Poor performance of the ${\text{cS}}_{pred}$ baseline shows that the distributions of a word in the source and target domains are different to the extent that the distributional features found using source domain distributions are inadequate. The two baselines ${\text{cS}}_{pred}$ and ${\text{cT}}_{pred}$ collectively motivate our proposal to learn a distribution prediction model from the source domain to the target. The improvements of Proposed over the previously proposed Filter are statistically significant in all domains except the Emails domain (denoted by $†$ in Table 2 according to the Binomial exact test at $95\mathrm{\%}$ confidence). However, the differences between the ${\text{cT}}_{pred}$ and Proposed methods are not statistically significant.

In Figure 1, we compare the Proposed cross-domain sentiment classification method (Section 4.2) against several baselines and the current state-of-the-art methods. The baselines NA, ${\text{cS}}_{\mathrm{𝐩𝐫𝐞𝐝}}$, and ${\text{cT}}_{\mathrm{𝐩𝐫𝐞𝐝}}$ are defined similarly as in Section 6.1. SST is the Sentiment Sensitive Thesaurus proposed by . SST creates a single distribution for a word using both source and target domain reviews, instead of two separate distributions as done by the Proposed method. SCL denotes the Structural Correspondence Learning method proposed by . SFA denotes the Spectral Feature Alignment method proposed by . SFA and SCL represent the current state-of-the-art methods for cross-domain sentiment classification. All methods are evaluated under the same settings, including train/test split, feature spaces, pivots, and classification algorithms so that any differences in performance can be directly attributable to their domain adaptability. For each domain, the accuracy obtained by a classifier trained using labeled data from that domain is indicated by a solid horizontal line in each sub-figure. This upper baseline represents the classification accuracy we could hope to obtain if we were to have labeled data for the target domain. Clopper-Pearson $95\mathrm{\%}$ binomial confidence intervals are superimposed on each vertical bar.

From Figure 1 we see that the Proposed method reports the best results in $8$ out of the $12$ domain pairs, whereas SCL, SFA, and ${\text{cS}}_{\mathrm{𝐩𝐫𝐞𝐝}}$ report the best results in other cases. Except for the D-E setting in which Proposed method significantly outperforms both SFA and SCL, the performance of the Proposed method is not statistically significantly different to that of SFA or SCL.

The selection of pivots is vital to the performance of SFA. However, unlike SFA, which requires us to carefully select a small subset of pivots (ca. less than $500$) using some heuristic approach, our Proposed method does not require any pivot selection. Moreover, SFA projects source domain reviews to a lower-dimensional latent space, in which a binary sentiment classifier is subsequently trained. At test time SFA projects a target review into this lower-dimensional latent space and applies the trained classifier. In contrast, our Proposed method predicts the distribution of a word in the target domain, given its distribution in the source domain, thereby explicitly translating the source domain reviews to the target. This property enables us to apply the proposed distribution prediction method to tasks other than sentiment analysis such as POS tagging where we must identify distributional features for individual words.

Unlike our distribution prediction method, which is unsupervised, SST requires labeled data for the source domain to learn a feature mapping between a source and a target domain in the form of a thesaurus. However, from Figure 1 we see that in $10$ out of the $12$ domain-pairs the Proposed method returns higher accuracies than SST.

To evaluate the overall effect of the number of singular vectors $k$ used in the SVD step, and the number of PLSR components $L$ used in Algorithm 3.2, we conduct two experiments. To evaluate the effect of the PLSR dimensions, we fixed $k=1000$ and measured the cross-domain sentiment classification accuracy over a range of $L$ values. As shown in Figure 2, accuracy remains stable across a wide range of PLSR dimensions. Because the time complexity of Algorithm 3.2 increases linearly with $L$, it is desirable that we select smaller $L$ values in practice.

To evaluate the effect of the SVD dimensions, we fixed $L=100$ and measured the cross-domain sentiment classification accuracy for different $k$ values as shown in Figure 3. We see an overall decrease in classification accuracy when $k$ is increased. Because the dimensionality of the source and target domain feature spaces is equal to $k$, the complexity of the least square regression problem increases with $k$. Therefore, larger $k$ values result in overfitting to the train data and classification accuracy is reduced on the target test data.

As an example of the distribution prediction method, in Table 3 we show the top $3$ similar distributional features $u$ in the books (source) domain, predicted for the electronics (target) domain word $w=\text{𝑙𝑖𝑔ℎ𝑡𝑤𝑒𝑖𝑔ℎ𝑡}$, by different similarity measures. Bigrams are indicted by a $+$ sign and the similarity scores of the distributional features are shown within brackets.

Using the source domain distributions for both $u$ and $w$ (i.e. $\mathrm{sim}\left(u_{\text{cS}},w_{\text{cS}}\right)$) produces distributional features that are specific to the books domain, or to the dominant adjectival sense of having no importance or influence. On the other hand, using target domain distributions for $u$ and $w$ (i.e. $\mathrm{sim}\left(u_{\text{cT}},w_{\text{cT}}\right)$) returns distributional features of the dominant nominal sense of lower in weight frequently associated with electronic devices. Simply using source domain distributions $u_{\text{cS}}$ (i.e. $\mathrm{sim}\left(u_{\text{cS}},w_{\text{cT}}\right)$) returns totally unrelated distributional features. This shows that word distributions in source and target domains are very different and some adaptation is required prior to computing distributional features.

Interestingly, we see that by using the distributions predicted by the proposed method (i.e. $\mathrm{sim}\left(\text{mat}M{u}_{\text{cS}},w_{\text{cT}}\right)$) we overcome this problem and find relevant distributional features from the source domain. Although for illustrative purposes we used the word lightweight, which occurs in both the source and the target domains, our proposed method does not require the source domain distribution $w_{\text{cS}}$ for a word $w$ in a target domain document. Therefore, it can find distributional features even for words occurring only in the target domain, thereby reducing the feature mismatch between the two domains.