Aspect Extraction with Automated Prior Knowledge Learning

After running LDA (or AKL) on each domain corpus, a set of topics is obtained. Each topic is a distribution over terms (or words), i.e., terms with their associated probabilities. Here, we use only the top terms with high probabilities. As discussed earlier, quality knowledge should be shared by topics across several domains. Thus, it is natural to exploit a frequency-based approach to discover frequent set of terms as quality knowledge. However, we need to deal with two issues.

[leftmargin=*]
1. 1.

   Generic aspects, such as price with aspect terms like cost and pricy, are shared by many (even all) product domains. But specific aspects such as screen, occur only in domains with products having them. It means that different aspects may have distinct frequencies. Thus, using a single frequency threshold in the frequency-based approach is not sufficient to extract both generic and specific aspects because the generic aspects will result in numerous spurious aspects [17].

2. 2.

   A term may have multiple senses in different domains. For example, light can mean “of little weight” or “something that makes things visible”. A good knowledge base should have the capacity of handling this ambiguity.

To deal with these two issues, we propose to discover knowledge in two stages: topic clustering and frequent pattern mining (FPM).

The purpose of clustering is to group raw topics from a topic model (LDA or AKL) into clusters. Each cluster contains semantically related topics likely to indicate the same real-world aspect. We then mine knowledge from each cluster using an FPM technique. Note that the multiple senses of a term can be distinguished by the semantic meanings represented by the topics in different clusters.

For clustering, we tried $k$-means and $k$-medoids [26], and found that $k$-medoids performs slightly better. One possible reason is that $k$-means is more sensitive to outliers. In our topic clustering, each data point is a topic represented by its top terms (with their probabilities normalized). The distance between two data points is measured by symmetrised KL-Divergence.

Given topics within each cluster, this step finds sets of terms that appear together in multiple topics, i.e., shared terms among similar topics across multiple domains. Terms in such a set are likely to belong to the same aspect. To find such sets of terms within each cluster, we use frequent pattern mining (FPM) [17], which is suited for the task. The probability of each term is ignored in FPM.

FPM is stated as follows: Given a set of transactions $\mathrm{T}$, where each transaction $t_i\in \mathrm{T}$ is a set of items from a global item set $\mathrm{I}$, i.e., $t_i\in \mathrm{I}$. In our context, $t_i$ is the topic vector comprising the top terms of a topic (no probability attached). $\mathrm{T}$ is the collection of all topics within a cluster and $\mathrm{I}$ is the set of all terms in $\mathrm{T}$. The goal of FPM is to find all patterns that satisfy some user-specified frequency threshold (also called minimum support count), which is the minimum number of times that a pattern should appear in $\mathrm{T}$. Such patterns are called frequent patterns. In our context, a pattern is a set of terms which have appeared multiple times in the topics within a cluster. Such patterns compose our knowledge base as shown below.

As the knowledge is extracted from each cluster individually, we represent our knowledge base as a set of clusters, where each cluster consists of a set of frequent 2-patterns mined using FPM, e.g.,

Cluster 1: {battery, life}, {battery, hour}, {battery, long}, {charge, long}

Cluster 2: {service, support}, {support, customer}, {service, customer}

Using two terms in a set is sufficient to cover the semantic relationship of the terms belonging to the same aspect. Longer patterns tend to contain more errors since some terms in a set may not belong to the same aspect as others. Such partial errors hurt performance in the downstream model.

Differing from most topic models based on topic-term distribution, AKL incorporates a latent cluster variable $𝒄$ to connect topics and terms. The plate notation of AKL is shown in Figure 2. The inputs of the model are $M$ documents, $T$ topics and $C$ clusters. Each document $m$ has $N_m$ terms. We model distribution $P\left(cluster\right|topic)$ as $\psi$ and distribution $P\left(term\right|topic,cluster)$ as $\phi$ with Dirichlet priors $\beta$ and $\gamma$ respectively. $P\left(topic\right|document)$ is modeled by $\theta$ with a Dirichlet prior $\alpha$. The terms in each document are assumed to be generated by first sampling a topic $z$, and then a cluster $c$ given topic $z$, and finally a term $w$ given topic $z$ and cluster $c$. This plate notation of AKL and its associated generative process are similar to those of MC-LDA [11]. However, there are three key differences.

[leftmargin=*]
1. 1.

   Our knowledge is automatically mined which may have errors (or noises), while the prior knowledge for MC-LDA is manually provided and assumed to be correct. As we will see in Section 6, using our knowledge, MC-LDA does not generate as coherent aspects as AKL.

2. 2.

   Our knowledge is represented as clusters. Each cluster contains a set of frequent 2-patterns with semantically correlated terms. They are different from must-sets used in MC-LDA.

3. 3.

   Most importantly, due to the use of the new form of knowledge, AKL’s inference mechanism (Gibbs sampler) is entirely different from that of MC-LDA (Section 5.2), which results in superior performances (Section 6). Note that the inference mechanism and the prior knowledge cannot be reflected in the plate notation for AKL in Figure 2.

In short, our modeling contributions are (1) the capability of handling more expressive knowledge in the form of clusters, (2) a novel Gibbs sampler to deal with inappropriate knowledge.

As the automatically learned prior knowledge may contain errors for a domain, AKL has to learn the usefulness of each piece of knowledge dynamically during inference. Instead of assigning weights to each piece of knowledge as a fixed prior in [10], we propose a new Gibbs sampler, which can dynamically balance the use of prior knowledge and the information in the corpus during the Gibbs sampling iterations.

We adopt a Blocked Gibbs sampler [51] as it improves convergence and reduces autocorrelation when the variables (topic $z$ and cluster $c$ in AKL) are highly related. For each term $w_i$ in each document, we jointly sample a topic $z_i$ and cluster $c_i$ (containing $w_i$) based on the conditional distribution in Gibbs sampler (will be detailed in Equation 4). To compute this distribution, instead of considering how well $z_i$ matches with $w_i$ only (as in LDA), we also consider two other factors:

[leftmargin=*]
1. 1.

   The extent $c_i$ corroborates $w_i$ given the corpus. By “corroborate”, we mean whether those frequent 2-patterns in $c_i$ containing $w_i$ are also supported by the actual information in the domain corpus to some extent (see the measure in Equation 1 below). If $c_i$ corroborates $w_i$ well, $c_i$ is likely to be useful, and thus should also provide guidance in determining $z_i$. Otherwise, $c_i$ may not be a suitable piece of knowledge for $w_i$ in the domain.

2. 2.

   Agreement between $c_i$ and $z_i$. By agreement we mean the degree that the terms (union of all frequent 2-patterns of $c_i$) in cluster $c_i$ are reflected in topic $z_i$. Unlike the first factor, this is a global factor as it concerns all the terms in a knowledge cluster.

For the first factor, we measure how well $c_i$ corroborates $w_i$ given the corpus based on co-document frequency ratio. As shown in [42], co-document frequency is a good indicator of term correlation in a domain. Following [42], we define a symmetric co-document frequency ratio as follows:

where $\left(w,w^{\prime }\right)$ refers to each frequent 2-pattern in the knowledge cluster $c_i$. $D\left(w,w^{\prime }\right)$ is the number of documents that contain both terms $w$ and $w^{\prime }$ and $D\left(w\right)$ is the number of documents containing $w$. A smoothing count of $1$ is added to avoid the ratio being $0$.

For the second factor, if cluster $c_i$ and topic $z_i$ agree, the intuition is that the terms in $c_i$ (union of all frequent $2$-patterns of $c_i$) should appear as top terms under $z_i$ (i.e., ranked top according to the term probability under $z_i$). We define the agreement using symmetrised KL-Divergence between the two distributions ($DIS{T}_c$ and $DIS{T}_z$) corresponding to $c_i$ and $z_i$ respectively. As there is no prior preference on the terms of $c_i$, we use the uniform distribution over all terms in $c_i$ for $DIS{T}_c$. For $DIS{T}_z$, as only top $20$ terms under $z_i$ are usually reliable, we use these top terms with their probabilities (re-normalized) to represent the topic. Note that a smoothing probability (i.e., a very small value) is also given to every term for calculating KL-Divergence. Given $DIS{T}_c$ and $DIS{T}_z$, the agreement is computed with:

The rationale of Equation 2 is that the lesser divergence between $DIS{T}_c$ and $DIS{T}_z$ implies the more agreement between $c_i$ and $z_i$.

We further employ the Generalized Pólya urn (GPU) model [40] which was shown to be effective in leveraging semantically related words [10, 11, 42]. The GPU model here basically states that assigning topic $z_i$ and cluster $c_i$ to term $w_i$ will not only increase the probability of connecting $z_i$ and $c_i$ with $w_i$, but also make it more likely to associate $z_i$ and $c_i$ with term $w^{\prime }$ where $w^{\prime }$ shares a 2-pattern with $w_i$ in $c_i$. The amount of probability increase is determined by matrix ${\mathrm{A}}_{\mathrm{c},{\mathrm{w}}^{\prime },\mathrm{w}}$ defined as:

where value $1$ controls the probability increase of $w$ by seeing $w$ itself, and $\sigma$ controls the probability increase of $w^{\prime }$ by seeing $w$. Please refer to [11] for more details.

Putting together Equations 1, 2 and 3 into a blocked Gibbs Sampler, we can define the following sampling distribution in Gibbs sampler so that it provides helpful guidance in determining the usefulness of the prior knowledge and in selecting the semantically coherent topic.

where $n^{-i}$ denotes the count excluding the current assignment of $z_i$ and $c_i$, i.e., ${𝒛}^{-i}$ and ${𝒄}^{-i}$. $n_{m,t}$ denotes the number of times that topic $t$ was assigned to terms in document $m$. $n_{t,c}$ denotes the times that cluster $c$ occurs under topic $t$. $n_{t,c,v}$ refers to the number of times that term $v$ appears in cluster $c$ under topic $t$. $\alpha$, $\beta$ and $\gamma$ are predefined Dirichlet hyperparameters.

Note that although the above Gibbs sampler is able to distinguish useful knowledge from wrong knowledge, it is possible that there is no cluster corroborates for a particular term. For every term $w$, apart from its knowledge clusters, we also add a singleton cluster for $w$, i.e., a cluster with one pattern $\left\{w,w\right\}$ only. When no knowledge cluster is applicable, this singleton cluster is used. As a singleton cluster does not contain any knowledge information but only the word itself, Equations 1 and 2 cannot be computed. For the values of singleton clusters for these two equations, we assign them as the averages of those values of all non-singleton knowledge clusters.

Dataset. We created a large dataset containing reviews from $36$ product domains or types from Amazon.com. The product domain names are listed in Table 1. Each domain contains $1,000$ reviews. This gives us $36$ domain corpora. We have made the dataset publically available at the website of the first author.

Pre-processing. We followed [11] to employ standard pre-processing like lemmatization and stopword removal. To have a fair comparison, we also treat each sentence as a document as in  [10, 11].

Parameter Settings. For all models, posterior estimates of latent variables were taken with a sampling lag of $20$ iterations in the post burn-in phase (first $200$ iterations for burn-in) with $2,000$ iterations in total. The model parameters were tuned on the development set in our pilot experiments and set to $\alpha =1$, $\beta =0.1$, $T=15$, and $\sigma =0.2$. Furthermore, for each cluster, $\gamma$ is set proportional to the number of terms in it. The other parameters for MC-LDA and GK-LDA were set as in their original papers. For parameters of AKL, we used the top $15$ terms for each topic in the clustering phrase. The number of clusters is set to the number of domains. We will test the sensitivity of these clustering parameters in Section 6.4. The minimum support count for frequent pattern mining was set empirically to $min\left(5,0.4\times \mathrm{\#}\mathrm{T}\right)$, where $\mathrm{\#}\mathrm{T}$ is the number of transactions (i.e., the number of topics from all domains) in a cluster.

Test Settings: We use two test settings as below:

[leftmargin=*]
1. 1.

   (Sections 6.2, 6.3 and 6.4) Test on the same corpora as those used in learning the prior knowledge. This is meaningful as the learning phrase is automatic and unsupervised (Figure 1). We call this self-learning-and-improvement.

2. 2.

   (Section 6.5) Test on new/unseen domain corpora after knowledge learning.

This sub-section evaluates the topics/aspects generated by each model based on Topic Coherence [42] in test setting 1. Traditionally, topic models have been evaluated using perplexity. However, perplexity on the held-out test set does not reflect the semantic coherence of topics and may be contrary to human judgments [9]. Instead, the metric Topic Coherence has been shown in  [42] to correlate well with human judgments. Recently, it has become a standard practice to use Topic Coherence for evaluation of topic models [3]. A higher Topic Coherence value indicates a better topic interpretability, i.e., semantically more coherent topics.

Figure 3 shows the average Topic Coherence of each model using knowledge learned at different learning iterations (Figure 1). For MC-LDA or GK-LDA, this is done by replacing AKL in lines 7 and 19 of Figure 1 with MC-LDA or GK-LDA. Each value is the average over all 36 domains. From Figure 3, we can observe the followings:

[leftmargin=*]
1. 1.

   AKL performs the best with the highest Topic Coherence values at all iterations. It is actually the best in all 36 domains. These show that AKL finds more interpretable topics than the baselines. Its values stabilize after iteration 3.

2. 2.

   Both GK-LDA and MC-LDA perform slightly better than LDA in iterations 1 and 2. MC-LDA does not handle wrong knowledge. This shows that the mined knowledge is of good quality. Although GK-LDA uses large word probability differences under a topic to detect wrong lexical knowledge, it is not as effective as AKL. The reason is that as the lexical knowledge is from general dictionaries rather than mined from relevant domain data, the words in a wrong piece of knowledge usually have a very large probability difference under a topic. However, our knowledge is mined from top words in related topics including topics from the current domain. The words in a piece of incorrect (or correct) knowledge often have similar probabilities under a topic. The proposed dynamic knowledge adjusting mechanism in AKL is superior.

Paired $t$-test shows that AKL outperforms all baselines significantly .

As our objective is to discover more coherent aspects, we recruited two human judges. Here we also use the test setting 1. Each topic is annotated as coherent if the judge feels that most of its top terms coherently represent a real-world product aspect; otherwise incoherent. For a coherent topic, each top term is annotated as correct if it reflects the aspect represented by the topic; otherwise incorrect. We labeled the topics of each model at learning iteration $1$ where the same pieces of knowledge (extracted from LDA topics at learning iteration 0) are provided to each model. After learning iteration 1, the gap between AKL and the baseline models tends to widen. To be consistent, the results later in Sections 6.4 and 6.5 also show each model at learning iteration 1. We also notice that after a few learning iterations, the topics from AKL model tend to have some resemblance across domains. We found that AKL with $2$ learning iterations achieved the best topics. Note that LDA cannot use any prior knowledge.

We manually labeled results from four domains, i.e., Camera, Computer, Headphone, and GPS. We chose Headphone as it has a lot of overlapping of topics with other domains because many electronic products use headphone. GPS was chosen because it does not have much topic overlapping with other domains as its aspects are mostly about Navigation and Maps. Domains Camera and Computer lay in between. We want to see how domain overlapping influences the performance of AKL. Cohen’s Kappa scores for annotator agreement are $0.918$ (for topics) and $0.872$ (for terms).

To measure the results, we compute $Precision\text{@}n$ (or $p\text{@}n$) based on the annotations, which was also used in [11, 44].

Figure 4 shows the $precision\text{@}n$ results for $n=5$ and $10$. We can see that AKL makes improvements in all $4$ domains. The improvement varies in domains with the most increase in Headphone and the least in GPS as Headphone overlaps more with other domains than GPS. Note that if a domain shares aspects with many other domains, its model should benefit more; otherwise, it is reasonable to expect lesser improvements. For the baselines, GK-LDA and MC-LDA perform similarly to LDA with minor variations, all of which are inferior to AKL. AKL’s improvements over other models are statistically significant based on paired $t$-test .

In terms of the number of coherent topics, AKL discovers one more coherent topic than LDA in Computer and one more coherent topic than GK-LDA and MC-LDA in Headphone. For the other domains, the numbers of coherent topics are the same for all models.

Table 2 shows an example aspect (battery) and its top $10$ terms produced by AKL and LDA for each domain to give a flavor of the kind of improvements made by AKL. The results for GK-LDA and MC-LDA are about the same as LDA (see also Figure 4). Table 2 focuses on the aspects generated by AKL and LDA. From Table 2, we can see that AKL discovers more correct and meaningful aspect terms at the top. Note that those terms marked in red and italicized are errors. Apart from Table 2, many aspects are dramatically improved by AKL, including some commonly shared aspects such as Price, Screen, and Customer Service.

This sub-section investigates the sensitivity of the clustering parameters of AKL (again in test setting $1$). The top sub-figure in Figure 5 shows the average Topic Coherence values versus the top $k$ terms per topic used in topic clustering (Section 4.1). The number of clusters is set to the number of domains (see below). We can observe that using $k=15$ top terms gives the highest value. This is intuitive as too few (or too many) top terms may generate insufficient (or noisy) knowledge.

The bottom sub-figure in Figure 5 shows the average Topic Coherence given different number of clusters. We fix the number of top terms per topic to $15$ as it yields the best result (see the top sub-figure in Figure 5). We can see that the performance is not very sensitive to the number of clusters. The model performs similarly for $30$ to $50$ clusters, with lower Topic Coherence for less than $30$ or more than $50$ clusters. The significance test indicates that using $30$, $40$, and $50$ clusters, AKL achieved significant improvements over all baseline models . With more domains, we should expect a larger number of clusters. However, it is difficult to obtain the optimal number of clusters. Thus, we empirically set the number of clusters to the number of domains in our experiments. Note that the number of clusters $\left(C\right)$ is expected to be larger than the number of topics in one domain $\left(T\right)$ because $C$ is for all domains while $T$ is for one particular domain.

We now evaluate AKL in test setting $2$, i.e., the automatically extracted knowledge $𝑲$ (Figure 1) is applied in new/unseen domains other than those in domains ${𝑫}_{𝑳}$ used in knowledge learning. The aim is to see how $𝑲$ can help modeling in an unseen domain. In this set of experiments, each domain is tested by using the learned knowledge from the rest $35$ domains. Figure 6 shows the average Topic Coherence of each model. The values are also averaged over the $36$ tested domains. We can see that AKL achieves the highest Topic Coherence value while LDA has the lowest. The improvements of AKL over all baseline models are significant with .