Learning Polylingual Topic Models from
Code-Switched Social Media Documents

Topic models [] have become standard tools for analyzing document collections, and topic analyses are quite common for social media []. Their popularity owes in part to their data driven nature, allowing them to adapt to new corpora and languages. In social media especially, there is a large diversity in terms of both the topic and language, necessitating the modeling of multiple languages simultaneously. A good candidate for multi-lingual topic analyses are polylingual topic models [], which learn topics for multiple languages, creating tuples of language specific distributions over monolingual vocabularies for each topic. Polylingual topic models enable cross language analysis by grouping documents by topic regardless of language.

Training of polylingual topic models requires parallel or comparable corpora: document tuples from multiple languages that discuss the same topic. While additional non-aligned documents can be folded in during training, the “glue” documents are required to aid in the alignment across languages. However, the ever changing vocabulary and topics of social media [] make finding suitable comparable corpora difficult. Standard techniques – such as relying on machine translation parallel corpora or comparable documents extracted from Wikipedia in different languages – fail to capture the specific terminology of social media. Alternate methods that rely on bilingual lexicons [] similarly fail to adapt to shifting vocabularies. The result: an inability to train polylingual models on social media.

In this paper, we offer a solution: utilize code-switched social media to discover correlations across languages. Social media is filled with examples of code-switching, where users switch between two or more languages, both in a conversation and even a single message []. This mixture of languages in the same context suggests alignments between words across languages through the common topics discussed in the context.

We learn from code-switched social media by extending the polylingual topic model framework to infer the language of each token and then automatically processing the learned topics to identify aligned topics. Our model improves both in terms of perplexity and a human evaluation, and we provide some example analyses of social media that rely on our learned topics.

Code-switched documents has received considerable attention in the NLP community. Several tasks have focused on identification and analysis, including mining translations in code-switched documents [], predicting code-switched points [], identifying code-switched tokens [], adding code-switched support to language models [], linguistic processing of code switched data [], corpus creation [], and computational linguistic analyses and theories of code-switching [].

Code-switching specifically in social media has also received some recent attention. trained a supervised token level language identification system for Spanish and English code-switched social media to study code-switching behaviors. mined translation spans for Chinese and English in code-switched documents to improve a translation system, relying on an existing translation model to aid in the identification and extraction task. In contrast to this work, we take an unsupervised approach, relying only on readily available document level language ID systems to utilize code-switched data. Additionally, our focus is not on individual messages, rather we aim to train a model that can be used to analyze entire corpora.

In this work we consider two types of code-switched documents: single messages and conversations, and two language pairs: Chinese-English and Spanish-English. Figure 1 shows an example of a code-switched Spanish-English conversation, in which three users discuss Mexico’s football team advancing to the Gold medal game in the 2012 Summer Olympics. In this conversation, some tweets are code-switched and some are in a single language. By collecting the entire conversation into a single document we provide the topic model with additional content. An example of a Chinese-English code-switched messages is given by :

{CJK}

UTF8gbsn watup Kenny Mayne!! - Kenny Mayne 最近这么样啊!!

Here a user switches between languages in a single message. We empirically evaluate our model on both conversations and messages. In the model presentation we will refer to both as “documents.”

To train a polylingual topic model on social media, we make two modifications to the model of : add a token specific language variable, and a process for identifying aligned topics.

First, polylingual topic models require parallel or comparable corpora in which each document has an assigned language. In the case of code-switched social media data, we require a per-token language variable. However, while document level language identification (LID) systems are common place, very few languages have per-token LID systems [].

To address the lack of available LID systems, we add a per-token latent language variable to the polylingual topic model. For documents that are not code-switched, we observe these variables to be the output of a document level LID system. In the case of code-switched documents, these variables are inferred during model inference.

Second, polylingual topic models assume the aligned topics are from parallel or comparable corpora, which implicitly assumes that a topics popularity is balanced across languages. Topics that show up in one language necessarily show up in another. However, in the case of social media, we can make no such assumption. The topics discussed are influenced by users, time, and location, all factors intertwined with choice of language. For example, English speakers will more likely discuss Olympic basketball while Spanish speakers football. There may be little or no documents on a given topic in one language, while they are plentiful in another. In this case, a polylingual topic model, which necessarily infers a topic-specific word distribution for each topic in each language, would learn two unrelated word distributions in two languages for a single topic. Therefore, naively using the produced topics as “aligned” across languages is ill-advised.

Our solution is to automatically identify aligned polylingual topics after learning by examining a topic’s distribution across code-switched documents. Our metric relies on distributional properties of an inferred topic across the entire collection.

To summarize, based on the model of we will learn:

• •

  For each topic, a language specific word distribution.

• •

  For each (code-switched) token, a language.

• •

  For each topic, an identification as to whether the topic captures an alignment across languages.

The first two goals are achieved by incorporating new hidden variables in the traditional polylingual topic model. The third goal requires an automated post-processing step. We call the resulting model Code-Switched LDA (csLDA). The generative process is as follows:

• •

  For each topic $z\in 𝒯$

  • –

    For each language $l\in ℒ$

    • *

      Draw word distribution ${\varphi }_z^l$$\sim$$Dir\left({\beta }^l\right)$

• •

  For each document $d\in 𝒟$:

  • –

    Draw a topic distribution ${\theta }_d\sim Dir\left(\alpha \right)$

  • –

    Draw a language distribution ${\psi }_d$$\sim$$Dir\left(\gamma \right)$

  • –

    For each token $i\in d$:

    • *

      Draw a topic $z_i\sim {\theta }_d$

    • *

      Draw a language $l_i\sim {\psi }_d$

    • *

      Draw a word $w_i\sim {\varphi }_z^l$

For monolingual documents, we fix $l_i$ to the LID tag for all tokens. Additionally, we use a single background distribution for each language to capture stopwords; a control variable $\pi$, which follows a Dirichlet distribution with prior parameterized by $\delta$, is introduced to decide the choice between background words and topic words following []¹¹Omitted from the generative process but shown in Fig. 2.. We use asymmetric Dirichlet priors [], and let the optimization process learn the hyperparameters. The graphical model is shown in Figure 2.

We used two datasets: a Sina Weibo Chinese-English corpus [] and a Spanish-English Twitter corpus.

We evaluated csLDA on the two datasets and evaluated each model using perplexity on held out data and human judgements. While our goal is to learn polylingual topics, we cannot compare to previous polylingual models since they require comparable data, which we lack. Instead, we constructed a baseline from LDA run on the entire dataset (no language information.) For each model, we measured the document completion perplexity [] on the held out data. We experimented with different numbers of topics ($𝒯$). Since csLDA duplicates topic distributions ($𝒯\times ℒ$) we used twice as many topics for LDA.

Figure 3 shows test perplexity for varying $𝒯$ and perplexity for the best setting of csLDA ($𝒯$ $=$$60$) and LDA ($𝒯$$=$$120$). The table lists both monolingual and code-switched test data; csLDA improves over LDA in almost every case, and across all values of $𝒯$. The background distribution (-bg) has mixed results for LDA, whereas for csLDA it shows consistent improvement. Table 1 shows some csLDA topics. While there are some mistakes, overall the topics are coherent and aligned.

We use the available per-token LID system [] for Spanish/English to justify csLDA’s ability to infer the hidden language variables. We ran csLDA-bg with $l_i$ set to the value provided by the LID system for code-switched documents (csLDA-bg with LID), which gives csLDA high quality LID labels. While we see gains for the code-switched data, overall the results for csLDA-bg and csLDA-bg with LID are similar, suggesting that the model can operate effectively even without a supervised per-token LID system.