Inferring User Political Preferences from Streaming Communications

Lets define an attributed, undirected graph $G=\left(V,E\right)$, where $V$ is a set of vertices and $E$ is a set of edges. Each vertex $v_i$ represents someone in a communication graph i.e., communicant: here a Twitter user. Each vertex is attributed with a feature vector $\overrightarrow{f}\left(v_i\right)$ which encodes communications e.g., tweets available for a given user. Each vertex is associated with a latent attribute $a\left(v_i\right)$, in our case it is binary $a\left(v_i\right)\in \left\{D,R\right\}$, where $D$ stands for Democratic and $R$ for Republican users. Each edge $e_{ij}\in E$ represents a connection between $v_i$ and $v_j$, $e_{ij}=\left(v_i,v_j\right)$ and defines different social circles between Twitter users e.g., follower $\left(f\right)$, friend $\left(b\right)$, user mention $\left(m\right)$, hashtag $\left(h\right)$, reply $\left(y\right)$ and retweet $\left(w\right)$. Thus, $E\in V^{\left(2\right)}\times \left\{f,b,h,m,w,y\right\}$. We denote a set of edges of a given type as ${\varphi }_r\left(E\right)$ for $r\in \left\{f,b,h,m,w,y\right\}$. We denote a set of vertices adjacent to $v_i$ by social circle type $r$ as $N_r\left(v_i\right)$ which is equivalent to $\left\{v_j\mid e_{ij}\in {\varphi }_r\left(E\right)\right\}$. Following Filippova (2012) we refer to $N_r\left(v_i\right)$ as $v_i$’s social circle, otherwise known as a neighborhood. In most cases, we only work with a sample of a social circle, denoted by $N_r^{\prime }\left(v_i\right)$ where $\left|N_r^{\prime }\left(v_i\right)\right|=k$ is its size for $v_i$.

Figure 1 presents an example of a social graph derived from Twitter. Notably, users from different social circles can be shared across the users of the same or different classes e.g., a user $v_j$ can be in both follower circle $v_j\in N_f\left(v_i\right),v_i\in D$ and retweet circle $v_j\in N_w\left(v_k\right),v_k\in R$.

We construct candidate-centric graph $G_{cand}$ by looking into following relationships between the users and Democratic or Republican candidates during the 2012 US Presidential election. In the Fall of 2012, leading up to the elections, we randomly sampled $n=516$ Democratic and $m=515$ Republican users. We labeled users as Democratic if they exclusively follow both Democratic candidates⁴⁴As of Oct 12, 2012, the number of followers for Obama, Biden, Romney and Ryan were 2m, 168k, 1.3m and 267k. – BarackObama and JoeBiden but do not follow both Republican candidates – MittRomney and RepPaulRyan and vice versa. We collectively refer to $D$ and $R$ as our “users of interest” for which we aim to predict political preference. For each such user we collect recent tweets and randomly sample their immediate $k=10$ neighbors from follower, friend, user mention, reply, retweet and hashtag social circles.

We construct a geo-centric graph $G_{geo}$ by collecting $n=135$ Democratic and $m=135$ Republican users from the Maryland, Virginia and Delaware region of the US with self-reported political preference in their biographies. Similar to the candidate-centric graph, for each user we collect recent tweets and randomly sample user social circles in the Fall of 2012. We collect this data to get a sample of politically less active users compared to the users from candidate-centric graph.

We also consider a $G_{ZLR}$ graph constructed from a dataset previously used for political affiliation classification [42]. This dataset consists of 200 Republican and 200 Democratic users associated with 925 tweets on average per user.⁵⁵The original dataset was collected in 2012 and has been recently released at http://icwsm.cs.mcgill.ca/. Political labels are extracted from http://www.wefollow.com as described by Pennacchiotti and Popescu (2011a). Each user has on average 6155 friends with 642 tweets per friend. Sharing restrictions and rate limits on Twitter data collection only allowed us to recreate a semblance of ZLR data⁶⁶This inability to perfectly replicate prior work based on Twitter is a recognized problem throughout the community of computational social science, arising from the data policies of Twitter itself, it is not specific to this work. – 193 Democratic and 178 Republican users with 1K tweets per user, and 20 neighbors of four types including follower, friends, user mention and retweet with 200 tweets per neighbor for each user of interest.

We first answer whether communications from user-local neighborhoods can help predict political preference for the user. To explore the contribution of different neighborhood types we learn static user and neighbor models on $G_{cand}$, $G_{geo}$ and $G_{ZLR}$ graphs. We also examine the ability of our static models to predict user political preferences in low-resource setting e.g., 5 tweets.

The existing models follow a standard setup when either user or neighbor tweets are available during train and test. For a static neighbor model we go beyond that, and train our the model on all data available per user, but only apply part of the data at the test time, pushing the boundaries of how little is truly required for classification. For example, we only use follower tweets for $G^{test}$, but we use tweets from all types of neighbors for $G^{train}$. Such setup will simulate different real-world prediction scenarios which have not been previously explored, to our knowledge e.g., when a user has a private profile or has not tweeted yet, and only user neighbor tweets are available.

We experiment with our static neighbor model defined in Eq.2 with the aim to:

1. 1.

   evaluate neighborhood size influence, we change the number of neighbors and try $n=\left[1,2,5,10\right]$ neighbor(s) per user;

2. 2.

   estimate neighbor content influence, we alternate the amount of content per neighbor and try $t=\left[5,10,15,25,50,100,200\right]$ tweets.

We perform 10-fold cross validation¹⁰¹⁰For each fold we split the data into 3 parts: 70% train, 10% development and 20% test. and run 100 random restarts for every $n$ and $t$ parameter combination. We compare our static neighbor and user models using the cost functions from Eq.3 and Eq.4. For all experiments we use LibLinear [9], integrated in the Jerboa toolkit [37]. Both models defined in Eq.1 and Eq.2 are learned using normalized count-based word ngram features extracted from either user or neighbor tweets.¹¹¹¹For brevity we omit reporting results for bigram and trigram features, since unigrams showed superior performance.

We evaluate our models with dynamic Bayesian updates on a continuous stream of communications over time as shown in Figure 2. Unlike static model experiments, we are not modeling the influence of the number of neighbors or the amount of content per neighbor. Here, we order user and neighbor communication streams by real world time of posting and measure changes in posterior probabilities over time. The main purpose of these experiments is to quantitatively evaluate (1) the number of tweets and (2) the amount of real world time it takes to observe enough evidence on Twitter to make reliable predictions.

We experiment with log-linear models defined in Eq. 1 and 2 and continuously estimate the posterior probabilities $P(R\mid T)$ as defined in Eq.6. We average the posterior probability results over the users in $G_{cand}$, $G_{geo}$ and $G_{ZLR}$ graphs. We train streaming models on an attribute balanced subset of tweets for each user $v_i$ excluding $v_i$’s tweets (or $v_i$’s neighbor tweets for a joint model). This setup is similar to leave-one-out classification. The classifier is learned using binary word ngram features extracted from user or user-neighbor communications. We prefer binary to normalized count-based features to overcome sparsity issues caused by making predictions on each tweet individually.

We investigate classification decision probabilities for our static user model ${\mathrm{\Phi }}_{v_i}$ by making predictions on a random set of 5 vs. 100 tweets per user. To our knowledge only limited work on personal analytics [5, 38] have performed this straight-forward comparison. For that purpose, we take a random partition containing 100 users of $G_{cand}$ graph and perform four independent classification experiments – two runs using 5 and two runs using 100 tweets per user.

Figure 3 demonstrates that more tweets during prediction time lead to higher accuracy by showing that more users with 100 tweets are correctly classified e.g., filled green markers in the right upper quadrant are true Republicans and in the left lower quadrant are true Democrats. Moreover, a lot of users with 100 tweets are close to 0.5 decision probability which suggests that the classifier is just uncertain rather then being completely off, e.g., misclassified Republican users with 5 tweets (not filled blue markers in the right lower quadrant) are close to 0. These results follow naturally from the underlying feature representation: having more tweets per user leads to a lower variance estimate of a target multinomial distribution. The more robustly this distribution is estimated (based on having more tweets) the more confident we should be in the classifier output.

Here we discuss the results for our static neighborhood model. We study the influence of the neighborhood type $r$ and size in terms of the number of neighbors $n$ and tweets $t$ per neighbor.

In Figure 4 we present accuracy results for $G_{cand}$ and $G_{geo}$ graphs. Following Eq.3 and 4, we spent an equal amount of resources to obtain 100 user tweets and 10 tweets from 10 neighbors. We annotate these ‘points of equal number of communications’ with a line on top marked with a corresponding number of user tweets.

We show that three of six social circles – friend, retweet and user-mention yield better accuracy compared to the user model for all graphs when $t\ge 250$. Thus, for effectively classifying a given user $v_i$ it is better to take 200 tweets each from 10 neighbors rather than 2,000 tweets from the user.

The best accuracy for $G_{cand}$ is 0.75 for friend, follower, retweet and user-mention neighborhoods which is 0.03 higher than the user baseline; for $G_{geo}$ is 0.67 for user-mention and 0.64 for retweet circles compared to 0.57 for the user model; for $G_{ZLR}$ is 0.863 for retweet and 0.849 for friend circles which is 0.11 higher that the user baseline. Finally, similarly to the results for the user model given in Figure 3, increasing the number of tweets per neighbor from 5 to 200 leads to a significant gain in performance for all neighborhood types.

In Figure 5 we present accuracy results to show neighborhood size influence on classification performance for $G_{geo}$ and $G_{cand}$ graphs. Our results demonstrate that even small changes to the neighborhood size $n$ lead to better performance which does not support the claims by Zamal et al. (2012). We demonstrate that increasing the size of the neighborhood leads to better performance across six neighborhood types. Friend, user mention and retweet neighborhoods yield the highest accuracy for all graphs. We observe that when the number of neighbors is $n=1$, the difference in accuracy across all neighborhood types is less significant but for $n\ge 2$ it becomes more significant.

Figures 6 and 6 demonstrate dynamic user model prediction results averaged over users from $G_{cand}$ and $G_{ZLR}$ graphs. Each figure outlines changes in sequential average probability estimates $p_{\mu }(R\mid T)$ for each individual self-authored tweet $t_k$ as defined in Eq. 6. The average probability estimates $p_{\mu }(R\mid T)$ are reported for every 5 tweets in a stream $T=\left(t_1,\mathrm{\dots }{t}_k\right)$ as $\frac{\sum {}_{n}P(R\mid t_k)}{n}$, where $n$ is the total number of users with the same attribute $R$ or $D$. We represent $p_{\mu }(R\mid T)$ as a box and whisker plot with the median, lower and upper quantiles to show the variance; the length of whiskers indicate lower and upper extreme values.

We find similar behavior across all three graphs. In particular, the posterior estimates converge faster when predicting Democratic than Republican users but it has been trained on an equal number of tweets per class. We observe that average posterior estimates $P_{\mu }(R\mid T)$ converge faster to 0 (Democratic) than to 1 (Republican) in Figures 6 and 6. It suggests that language of Democrats is more expressive of their political preference than language of Republicans. For example, frequent politically influenced terms used widely by Democratic users include faith4liberty, constitutionally, pass, vote2012, terroristic.

The variance for average posterior estimates decreases when the number of tweets increases for all three datasets. Moreover, we detect that $P_{\mu }\left(R\right|T)$ estimates for users in $G_{cand}$ converge 2-3 times faster in terms of number of tweets than for users in $G_{ZLR}$. The lowest convergence is detected for $G_{geo}$ where after $t_k=250$ tweets the average posterior estimate $P_{\mu }(R\mid t_k)=0.904\pm 0.044$ and $P_{\mu }(D\mid t_k)=0.861\pm 0.008$. It means that users in $G_{cand}$ are more politically vocal compared to users in $G_{ZLR}$ and $G_{geo}$. As a result, less active users in $G_{geo}$ just need more than 250 tweets to converge to a true 0 or 1 class. These results are coherent with the outcomes for our static models shown in Figures 4 and 5. These findings further confirm that differences in performance are caused by various biases present in the data due to distinct sampling and annotation approaches.

Figure 7a and 7b illustrate the amount of time required for the user model to infer political preferences estimated for 1,031 users in $G_{cand}$ and 371 users in $G_{ZLR}$. The amount of time needed can be evaluated for different accuracy levels e.g., 0.75 and 0.95. Thus, with 75% accuracy we classify:

• •

  100 ($\sim$20%) Republican users in 3.6 hours and Democratic users in 2.2 hours for $G_{cand}$;

• •

  100 ($\sim$56%) $R$ users in 20 weeks and 100 ($\sim$52%) $D$ users in 8.9 weeks for $G_{ZLR}$ which is 800 times longer that for $G_{cand}$;

• •

  100 ($\sim$75%) $R$ users in 12 weeks and 80 ($\sim$60%) $D$ users in 19 weeks for $G_{geo}$.

Such extreme divergences in the amount of time required for classification across all graphs should be of strong interest to researchers concerned with latent attribute prediction tasks because Twitter users produce messages with extremely different frequencies. In our case, users in $G_{ZLR}$ tweet approximately 800 times less frequently than users in $G_{cand}$.

We estimate dynamic posterior updates from a joint stream of user and neighbor communications in $G_{geo}$, $G_{cand}$ and $G_{ZLR}$ graphs. To make a fair comparison with a streaming user model, we start with the same user tweet $t_0\left(v_i\right)$. Then instead of waiting for the next user tweet we rely on any neighbor tweets that appear until the user produces the next tweet $t_1\left(v_i\right)$. We rely on communications from four types of neighbors such as friends, followers, retweets and user mentions.

The convergence rate for the average posterior probability estimates $P_{\mu }\left(R\right|T)$ depending on the number of tweets is similar to the user model results presented in Figure 6. However, for $G_{geo}$ the variance for $P_{\mu }\left(R\right|T)$ is higher for Democratic users; for $G_{ZLR}$ $P_{\mu }\left(R\right|T)\to 1$ for Republicans in less than 110 tweets which is $\mathrm{\Delta }t=40$ tweets faster than the user model; for $G_{cand}$ the convergence for both $P_{\mu }\left(R\right|T)\to 1$ and $P_{\mu }\left(D\right|T)\to 0$ is not significantly different than the user model.

Figures 7c and 7d show the amount of time required for a joint user-neighbor model to infer political preferences estimated for users in $G_{cand}$ and $G_{ZLR}$. We find that with 75% accuracy we can classify 100 users for:

• •

  $G_{cand}$: Republican users in 23 minutes and Democratic users in 10 minutes;

• •

  $G_{ZLR}$: $R$ users in 3.2 weeks and $D$ users in 1.1 weeks which is 7 times faster on average across attributes than for the user model;

• •

  $G_{geo}$: $R$ users in 1.2 weeks and $D$ users in 3.5 weeks which is on average 6 times faster across attributes than for the user model.

Similar or better $P_{\mu }\left(R\right|T)$ convergence in terms of the number of tweets and, especially, in the amount of time needed for user and user-neighbor models further confirms that neighborhood content is useful for political preference prediction. Moreover, communications from a joint stream allow to make an inference up to 7 times faster.