A Novel Content Enriching Model for Microblog Using News Corpus

We do topic analysis for $EK$ using LDA estimation [Blei et al.2003] in this section and we choose LDA as the topic analysis model because of its broadly proved effectivity and ease of understanding.

In LDA, each document has a distribution over all topics $P\left(z_k\right|d_j)$, and each topic has a distribution over all words $P\left(w_i\right|z_k)$, where $z_k$, $d_j$ and $w_i$ represent the topic, document and word respectively. The optimization problem is formulated as maximizing the log likelihood on the corpus:

In this formulation, $X_{ij}$ represents the term frequency of word $w_i$ in document $d_j$. $P\left(z_k\right|d_j)$ and $P\left(w_i\right|z_k)$ are parameters to be inferred, corresponding to the topic distribution of each document and the word distribution of each topic respectively. Estimating parameters for LDA by directly and exactly maximizing the likelihood of the corpus in (1) is intractable, so we use Gibbs Sampling for estimation.

After performing LDA model ($K$ topics) estimation on $EK$, we obtain the topic distributions of document $d_j^e$ ($j=1,\mathrm{\dots },M^e$), denoted as $P\left(z_k^e\right|d_j^e)$ ($k=1,\mathrm{\dots },K$), and the word distribution of topic $z_k^e$ ($k=1,\mathrm{\dots },K$), denoted as $P\left(w_i^e\right|z_k^e)$ ($i=1,\mathrm{\dots },N^e$). Step (b) greatly relies on the word distributions of topics we have obtained here.

In this section, we infer the topic distribution of each microblog. Because of the assumption that microblogs share the same topics with external corpus, the “topic distribution” here refers to a distribution over all topics on $EK$.

Differing from step (a), the method used for topic inference for microblogs is not directly running LDA estimation on microblog collection but following the topics from external knowledge to ensure topic consistence. We employ the word distributions of topics obtained from step (a), i.e. $P\left(w_i^e\right|z_k^e)$, and formulate the optimization problem in a similar form to Formula (1) as follows:

where ${\underset{¯}{X}}_{ij}$ represents the term frequency of word $w_i^e$ in microblog $d_j^m$, and $P\left(z_k^e\right|d_j^m)$ denote the distribution of microblog $d_j^m$ over all topics on $EK$. Obviously most ${\underset{¯}{X}}_{ij}$ are zero and we ignore those words that do not appear in $V^e$.

Compared with the original LDA optimization problem (1), the topic inference problem for microblog (2) follows the idea of document generation process, but replaces topics to be estimated with known topics from other corpus. As a result, parameters to be inferred are only the topic distribution of every microblog.

It is noteworthy that since the word distribution of every topic $P\left(w_i^e\right|z_k^e)$ is known, Formula (2) can be further solved by separating it into $M^m$ subproblems:

These $M^m$ subproblems correspond to the $M^m$ microblogs and can be easily proved convexity. After solving them, we obtain the topic distributions of microblog $d_j^m$ ($j=1,\mathrm{\dots },M^m$), denoted as $P\left(z_k^e\right|d_j^m)$ ($k=1,\mathrm{\dots },K$).

To enrich the content of every microblog, we select relevant words from external knowledge in this section.

Based on the results of step (a)&(b), we calculate the word distributions of microblogs as follows:

where $P\left(w_i^e\right|d_j^m)$ represents the probability that word $w_i^e$ will appear in microblog $d_j^m$. In other words, though some words may not actually appears in a microblog, there is still a probability that it is highly relevant to the microblog. Intuitively, this probability indicates the strength of association between a word and a microblog. The word distribution of every microblog is based on topic analysis and its accuracy relies heavily on the accuracy of topic inference in step (b). In fact, the more words a microblog includes, the more accurate its topic inference will be, and this can be regarded as an explanation of the low efficiency of data sparseness problem.

For microblog $d_j^m$, we sort all words by $P\left(w_i^e\right|d_j^m)$ in descending order. Having known the top $L$ relevant words according to the result of sorting, we redefine the “term frequency” of every word after adding these $L$ words to microblog $d_j^m$ as additional content. Supposing these $L$ words are $w_{j1}^e,w_{j2}^e,\mathrm{\dots },w_{jL}^e$, the revised term frequency of word $w\in \left\{w_{j1}^e,\mathrm{\dots },w_{jL}^e\right\}$ is defined as follows:

where $RTF(\cdot )$ is the revised term frequency.

As the Equation (5) shows, the revised term frequency of every word is proportional to probability $P\left(w_i\right|d_j^m)$ rather than a constant.

So far, we can add these $L$ words and their revised term frequency as additional information to microblog $d_j^m$. The revised term frequency plays the same role as TF in common text representation vector, so we calculate the TFIDF of the added words as:

Note that $IDF\left(w\right)$ is changed as arrival of new words for each microblog. The TFIDF vector of a microblog with additional words is called enhanced vector.

To evaluate our method, we build our own datasets. We crawl 95028 Chinese news reports from Sina News website^††Sina News: http://news.sina.com.cn/, segment them, and remove stop words and rare words. After preprocessing, these news documents are used as external knowledge. As for microblog, we crawl a number of microblogs from Sina Weibo^††Sina Weibo: http://www.weibo.com/, and ask unbiased assessors to manually classify them into 9 categories following the column setting of Sina News. After the manual classification, we remove short microblogs (less than $10$ words), usernames, links and some special characters, then we segment them and remove rare words as well. Finally, we get 1671 classified microblogs as our microblog dataset. The size of each category is shown in Table 1.

There are some important details of our implementation. In step (a) of Section 3.1 we estimate LDA model using GibbsLDA++^††GibbsLDA++: http://gibbslda.sourceforge.net, a C/C++ implementation of LDA using Gibbs Sampling. In step (b) of Section3.2, OPTI toolbox^††OPTI Toolbox: http://www.i2c2.aut.ac.nz/Wiki/OPTI/ on Matlab is used to help solve the convex problems. In the classification tasks shown below, we use LibSVM^††SVM.NET: http://www.matthewajohnson.org/software
/svm.html as classifier and perform ten-fold cross validation to evaluate the classification accuracy.

In this section, we report the average precision of each method as shown in Table 2. The enhanced vector is the representation generated by our method. Two baselines are TFIDF vector [Jones1972] and boolean vector (word occurrence) of the original microblog. In the table, our method increases the classification accuracy from 75.52% to 84.53% when considering additional information, which means our method indeed improves the representation of microblogs.

In Table 3, we select several cases consisting of microblogs and their top relevant words .

In the first case, we successfully find the country name according to its leader’s name and limited information in the sentence. Other related countries and events are also selected by our model as they often appear together in news. In the other case, relevant words are among the most frequently used words in news and have close semantic relations with the microblogs in certain aspects.

As we can see, based on topic analysis, our model shows strong ability of mining relevant words. Other cases show that the model can be further improved by removing the noisy and meaningless ones among added words.