A Piece of My Mind: A Sentiment Analysis Approach
for Online Dispute Detection

As the web has grown in popularity and scope, so has the promise of collaborative information environments for the joint creation and exchange of knowledge [11, 18]. Wikipedia, a wiki-based online encyclopedia, is arguably the best example: its distributed editing environment allows readers to collaborate as content editors and has facilitated the production of over four billion articles¹¹http://en.wikipedia.org of surprisingly high quality [6] in English alone since its debut in 2001.

Existing studies of collaborative knowledge systems have shown, however, that the quality of the generated content (e.g. an encyclopedia article) is highly correlated with the effectiveness of the online collaboration [12, 14]; fruitful collaboration, in turn, inevitably requires dealing with the disputes and conflicts that arise [13]. Unfortunately, human monitoring of the often massive social media and collaboration sites to detect, much less mediate, disputes is not feasible.

In this work, we investigate the heretofore novel task of dispute detection in online discussions. Previous work in this general area has analyzed dispute-laden content to discover features correlated with conflicts and disputes [13]. Research focused primarily on cues derived from the edit history of the jointly created content (e.g. the number of revisions, their temporal density [13, 23]) and relied on small numbers of manually selected discussions known to involve disputes. In contrast, we investigate methods for the automatic detection, i.e. prediction, of discussions involving disputes. We are also interested in understanding whether, and which, linguistic features of the discussion are important for dispute detection.

Drawing inspiration from studies of human mediation of online conflicts (e.g. Billings and Watts (2010), Kittur et al. (2007), Kraut and Resnick (2012)), we hypothesize that effective methods for dispute detection should take into account the sentiment and opinions expressed by participants in the collaborative endeavor. As a result, we propose a sentiment analysis approach for online dispute detection that identifies the sequence of sentence-level sentiments (i.e. very negative, negative, neutral, positive, very positive) expressed during the discussion and uses them as features in a classifier that predicts the dispute/non-dispute label for the discussion as a whole. Consider, for example, the snippet in Figure 1 from the Wikipedia Talk page for the article on Philadelphia; it discusses the choice of a picture for the article’s “infobox”. The sequence of almost exclusively negative statements provides evidence of a dispute in this portion of the discussion.

Unfortunately, sentence-level sentiment tagging for this domain is challenging in its own right due to the less formal, often ungrammatical, language and the dynamic nature of online conversations. “Really, grow up” (segment 3) should presumably be tagged as a negative sentence as should the sarcastic sentences “Sounds good?” (in the same turn) and “congrats” and “thank you” (in segment 2). We expect that these, and other, examples will be difficult for the sentence-level classifier unless the discourse context of each sentence is considered. Previous research on sentiment prediction for online discussions, however, focuses on turn-level predictions [7, 24].²²A notable exception is Hassan et al. (2010), which identifies sentences containing “attitudes” (e.g. opinions), but does not distinguish them w.r.t. sentiment. Context information is also not considered. As the first work that predicts sentence-level sentiment for online discussions, we investigate isotonic Conditional Random Fields (CRFs) [16] for the sentiment-tagging task as they preserve the advantages of the popular CRF-based sequential tagging models [15] while providing an efficient mechanism for encoding domain knowledge — in our case, a sentiment lexicon — through isotonic constraints on model parameters.

We evaluate our dispute detection approach using a newly created corpus of discussions from Wikipedia Talk pages (3609 disputes, 3609 non-disputes).³³The talk page associated with each article records conversations among editors about the article content and allows editors to discuss the writing process, e.g. planning and organizing the content. We find that classifiers that employ the learned sentiment features outperform others that do not. The best model achieves a very promising F1 score of 0.78 and an accuracy of 0.80 on the Wikipedia dispute corpus. To the best of our knowledge, this represents the first computational approach to automatically identify online disputes on a dataset of scale.

We construct the first dispute detection corpus to date; it consists of dispute and non-dispute discussions from Wikipedia Talk pages.

Step 1: Get Talk Pages of Disputed Articles. Wikipedia articles are edited by different editors. If an article is observed to have disputes on its talk page, editors can assign dispute tags to the article to flag it for attention. In this research, we are interested in talk pages whose corresponding articles are labeled with the following tags: disputed, totallydisputed, disputed-section, totallydisputed-section, pov. The tags indicate that an article is disputed, or the neutrality of the article is disputed (pov).

We use the 2013-03-04 Wikipedia data dump, and extract talk pages for articles that are labeled with dispute tags by checking the revision history. This results in 19,071 talk pages.

Step 2: Get Discussions with Disputes. Dispute tags can also be added to talk pages themselves. Therefore, in addition to the tags mentioned above, we also consider the “Request for Comment” (rfc) tag on talk pages. According to Wikipedia⁴⁴http://en.wikipedia.org/wiki/Wikipedia:Requests_for_comment, rfc is used to request outside opinions concerning the disputes.

3609 discussions are collected with dispute tags found in the revision history. We further classify dispute discussions into three subcategories: Controversy, Request for Comment (RFC), and Resolved based on the tags found in discussions (see Table 1). The numbers of discussions for the three types are 42, 3484, and 105, respectively. Note that dispute tags only appear in a small number of articles and talk pages. There may exist other discussions with disputes.

Step 3: Get Discussions without Disputes. Likewise, we collect non-dispute discussions from pages that are never tagged with disputes. We consider non-dispute discussions with at least 3 distinct speakers and 10 turns. 3609 discussions are randomly selected with this criterion. The average turn numbers for dispute and non-dispute discussions are 45.03 and 22.95, respectively.

This section describes our sentence-level sentiment tagger, from which we construct features for dispute detection (Section 4).

Consider a discussion comprised of sequential turns; each turn consists of a sequence of sentences. Our model takes as input the sentences $𝐱=\left\{x_1,\mathrm{\cdots },x_n\right\}$ from a single turn, and outputs the corresponding sequence of sentiment labels $𝐲=\left\{y_1,\mathrm{\cdots },y_n\right\}$, where $y_i\in 𝒪,𝒪=\left\{\mathrm{NN},\mathrm{N},\mathrm{O},\mathrm{P},\mathrm{PP}\right\}$. The labels in $𝒪$ represent very negative (NN), negative (N), neutral (O), positive (P), and very positive (PP), respectively.

Given that traditional Conditional Random Fields (CRFs) [15] ignore the ordinal relations among sentiment labels, we choose isotonic CRFs [16] for sentence-level sentiment analysis as they can enforce monotonicity constraints on the parameters consistent with the ordinal structure and domain knowledge (e.g. word-level sentiment conveyed via a lexicon). Concretely, we take a lexicon $ℳ={ℳ}_p\cup {ℳ}_n$, where ${ℳ}_p$ and ${ℳ}_n$ are two sets of features (usually words) identified as strongly associated with positive and negative sentiment. Assume ${\mu }_{⟨\sigma ,w⟩}$ encodes the weight between label $\sigma$ and feature $w$, for each feature $w\in {ℳ}_p$; then the isotonic CRF enforces $\sigma \le {\sigma }^{\prime }\Rightarrow {\mu }_{⟨\sigma ,w⟩}\le {\mu }_{⟨{\sigma }^{\prime },w⟩}$. For example, when “totally agree” is observed in training, parameter ${\mu }_{⟨\mathrm{PP},\mathrm{totally}\mathrm{agree}⟩}$ is likely to increase. Similar constraints are defined on ${ℳ}_n$.

Our lexicon is built by combining MPQA [22], General Inquirer [19], and SentiWordNet [3] lexicons. Words with contradictory sentiments are removed. We use the features in Table 2 for sentiment prediction.

Syntactic/Semantic Features. We have two versions of dependency relation features, the original form and a form that generalizes a word to its POS tag, e.g. “nsubj(wrong, you)” is generalized to “nsubj(ADJ, you)” and “nsubj(wrong, PRP)”.

Discourse Features. We extract the initial unigram, bigram, and trigram of each utterance as discourse features [9].

Sentiment Features. We gather connectives from the Penn Discourse TreeBank [17] and combine them with any sentiment word that precedes or follows it as new features. Sentiment dependency relations are the dependency relations that include a sentiment word. We replace those words with their polarity equivalents. For example, relation “nsubj(wrong, you)” becomes “nsubj(SentiWord${}_{n\mathtt{}e\mathtt{}g}$, you)”.

We present a sentiment analysis-based approach to online dispute detection. We create a large-scale dispute corpus from Wikipedia Talk pages to study the problem. A sentiment prediction model based on isotonic CRFs is proposed to output sentiment labels at the sentence-level. Experiments on our dispute corpus also demonstrate that classifiers trained with sentiment tagging features outperform others that do not.

Acknowledgments We heartily thank the Cornell NLP Group, the reviewers, and Yiye Ruan for helpful comments. We also thank Emily Bender and Mari Ostendorf for providing the AAWD dataset. This work was supported in part by NSF grants IIS-0968450 and IIS-1314778, and DARPA DEFT Grant FA8750-13-2-0015. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of NSF, DARPA or the U.S. Government.