Opinion Mining on YouTube

We enrich the traditional bag-of-word representation with features from a sentiment lexicon and features quantifying the negation present in the comment. Our model (FVEC) encodes each document using the following feature groups:

- word n-grams: we compute unigrams and bigrams over lower-cased word lemmas where binary values are used to indicate the presence/absence of a given item.

- lexicon: a sentiment lexicon is a collection of words associated with a positive or negative sentiment. We use two manually constructed sentiment lexicons that are freely available: the MPQA Lexicon [29] and the lexicon of Hu and Liu (2004). For each of the lexicons, we use the number of words found in the comment that have positive and negative sentiment as a feature.

- negation: the count of negation words, e.g., {don’t, never, not, etc.}, found in a comment.²²The list of negation words is adopted from http://sentiment.christopherpotts.net/lingstruc.html Our structural representation (defined next) enables a more involved treatment of negation.

- video concept: cosine similarity between a comment and the title/description of the video. Most of the videos come with a title and a short description, which can be used to encode the topicality of each comment by looking at their overlap.

We go beyond traditional feature vectors by employing structural models (STRUCT), which encode each comment into a shallow syntactic tree. These trees are input to tree kernel functions for generating structural features. Our structures are specifically adapted to the noisy user-generated texts and encode important aspects of the comments, e.g., words from the sentiment lexicons, product concepts and negation words, which specifically targets the sentiment and comment type classification tasks.

In particular, our shallow tree structure is a two-level syntactic hierarchy built from word lemmas (leaves) and part-of-speech tags that are further grouped into chunks (Fig. 1). As full syntactic parsers such as constituency or dependency tree parsers would significantly degrade in performance on noisy texts, e.g., Twitter or YouTube comments, we opted for shallow structures, which rely on simpler and more robust components: a part-of-speech tagger and a chunker. Moreover, such taggers have been recently updated with models [18, 5] trained specifically to process noisy texts showing significant reductions in the error rate on user-generated texts, e.g., Twitter. Hence, we use the CMU Twitter pos-tagger [5, 13] to obtain the part-of-speech tags. Our second component – chunker – is taken from [18], which also comes with a model trained on Twitter data³³The chunker from [18] relies on its own POS tagger, however, in our structural representations we favor the POS tags from the CMU Twitter tagger and take only the chunk tags from the chunker. and shown to perform better on noisy data such as user comments.

To address the specifics of OM tasks on YouTube comments, we enrich syntactic trees with semantic tags to encode: (i) central concepts of the video, (ii) sentiment-bearing words expressing positive or negative sentiment and (iii) negation words. To automatically identify concept words of the video we use context words (tokens detected as nouns by the part-of-speech tagger) from the video title and video description and match them in the tree. For the matched words, we enrich labels of their parent nodes (part-of-speech and chunk) with the PRODUCT tag. Similarly, the nodes associated with words found in the sentiment lexicon are enriched with a polarity tag (either positive or negative), while negation words are labeled with the NEG tag. It should be noted that vector-based (FVEC) model relies only on feature counts whereas the proposed tree encodes powerful contextual syntactic features in terms of tree fragments. The latter are automatically generated and learned by SVMs with expressive tree kernels.

For example, the comment in Figure 1 shows two positive and one negative word from the sentiment lexicon. This would strongly bias the FVEC sentiment classifier to assign a positive label to the comment. In contrast, the STRUCT model relies on the fact that the negative word, destroy, refers to the PRODUCT (xoom) since they form a verbal phase (VP). In other words, the tree fragment: [S [negative-VP [negative-V [destroy]] [PRODUCT-NP [PRODUCT-N [xoom]]]] is a strong feature (induced by tree kernels) to help the classifier to discriminate such hard cases. Moreover, tree kernels generate all possible subtrees, thus producing generalized (back-off) features, e.g., [S [negative-VP [negative-V [destroy]] [PRODUCT-NP]]]] or [S [negative-VP [PRODUCT-NP]]]].

We perform OM on YouTube using supervised methods, e.g., SVM. Our goal is to learn a model to automatically detect the sentiment and type of each comment. For this purpose, we build a multi-class classifier using the one-vs-all scheme. A binary classifier is trained for each of the classes and the predicted class is obtained by taking a class from the classifier with a maximum prediction score. Our back-end binary classifier is SVM-light-TK⁴⁴http://disi.unitn.it/moschitti/Tree-Kernel.htm, which encodes structural kernels in the SVM-light [7] solver. We define a novel and efficient tree kernel function, namely, Shallow syntactic Tree Kernel (SHTK), which is as expressive as the Partial Tree Kernel (PTK) [10] to handle feature engineering over the structural representations of the STRUCT model. A polynomial kernel of degree 3 is applied to feature vectors (FVEC).

Combining structural and vector models. A typical kernel machine, e.g., SVM, classifies a test input $𝒙$ using the following prediction function: $h\left(𝒙\right)={\sum }_i{\alpha }_i{y}_{i}K\left(𝒙,{𝒙}_i\right)$, where ${\alpha }_i$ are the model parameters estimated from the training data, $y_i$ are target variables, ${𝒙}_i$ are support vectors, and $K(\cdot ,\cdot )$ is a kernel function. The latter computes the similarity between two comments. The STRUCT model treats each comment as a tuple $𝒙=⟨𝑻,𝒗⟩$ composed of a shallow syntactic tree $𝑻$ and a feature vector $𝒗$. Hence, for each pair of comments ${𝒙}_1$ and ${𝒙}_2$, we define the following comment similarity kernel:

where $K_{\text{TK}}$ computes SHTK (defined next), and $K_{\text{v}}$ is a kernel over feature vectors, e.g., linear, polynomial, Gaussian, etc.

Shallow syntactic tree kernel. Following the convolution kernel framework, we define the new SHTK function from Eq. 1 to compute the similarity between tree structures. It counts the number of common substructures between two trees $T_1$ and $T_2$ without explicitly considering the whole fragment space. The general equations for Convolution Tree Kernels is:

where $N_{T_1}$ and $N_{T_2}$ are the sets of the $T_1$’s and $T_2$’s nodes, respectively and $\mathrm{\Delta }\left(n_1,n_2\right)$ is equal to the number of common fragments rooted in the $n_1$ and $n_2$ nodes, according to several possible definition of the atomic fragments.

To improve the speed computation of $TK$, we consider pairs of nodes $\left(n_1,n_2\right)$ belonging to the same tree level. Thus, given $H$, the height of the STRUCT trees, where each level $h$ contains nodes of the same type, i.e., chunk, POS, and lexical nodes, we define SHTK as the following⁵⁵To have a similarity score between 0 and 1, a normalization in the kernel space, i.e. $\frac{SHTK\left(T_1,T_2\right)}{\sqrt{SHTK\left(T_1,T_1\right)\times SHTK\left(T_2,T_2\right)}}$ is applied.:

where $N_{T_1}^h$ and $N_{T_2}^h$ are sets of nodes at height $h$.

The above equation can be applied with any $\mathrm{\Delta }$ function. To have a more general and expressive kernel, we use $\mathrm{\Delta }$ previously defined for PTK. More formally: if $n_1$ and $n_2$ are leaves then $\mathrm{\Delta }\left(n_1,n_2\right)=\mu \lambda \left(n_1,n_2\right)$; else $\mathrm{\Delta }\left(n_1,n_2\right)=$

where $\lambda ,\mu \in \left[0,1\right]$ are decay factors; the large sum is adopted from a definition of the subsequence kernel [22] to generate children subsets with gaps, which are then used in a recursive call to $\mathrm{\Delta }$. Here, $c_{n_1}\left(i\right)$ is the $i^{th}$ child of the node $n_1$; ${\overrightarrow{I}}_1$ and ${\overrightarrow{I}}_2$ are two sequences of indexes that enumerate subsets of children with gaps, i.e., $\overrightarrow{I}=(i_1,i_2,..,|I\left|\right)$, with ; and $d\left({\overrightarrow{I}}_1\right)={\overrightarrow{I}}_{1l\left({\overrightarrow{I}}_1\right)}-{\overrightarrow{I}}_{11}+1$ and $d\left({\overrightarrow{I}}_2\right)={\overrightarrow{I}}_{2l\left({\overrightarrow{I}}_2\right)}-{\overrightarrow{I}}_{21}+1$, which penalizes subsequences with larger gaps.

It should be noted that: firstly, the use of a subsequence kernel makes it possible to generate child subsets of the two nodes, i.e., it allows for gaps, which makes matching of syntactic patterns less rigid. Secondly, the resulting SHTK is essentially a special case of PTK [10], adapted to the shallow structural representation STRUCT (see Sec. 3.2). When applied to STRUCT trees, SHTK exactly computes the same feature space as PTK, but in faster time (on average). Indeed, SHTK required to be only applied to node pairs from the same level (see Eq. 3), where the node labels can match – chunk, POS or lexicals. This reduces the time for selecting the matching-node pairs carried out in PTK [10, 11]. The fragment space is obviously the same, as the node labels of different levels in STRUCT are different and will not be matched by PTK either.

Finally, given its recursive definition in Eq. 3 and the use of subsequence (with gaps), SHTK can derive useful dependencies between its elements. For example, it will generate the following subtree fragments: [positive-NP [positive-A N]], [S [negative-VP [negative-V [destroy]] [PRODUCT-NP]]]] and so on.

Sentiment classification. We treat each comment as expressing positive, negative or neutral sentiment. Hence, the task is a three-way classification.

Type classification. One of the challenging aspects of sentiment analysis of YouTube data is that the comments may express the sentiment not only towards the product shown in the video, but also the video itself, i.e., users may post positive comments to the video while being generally negative about the product and vice versa. Hence, it is of crucial importance to distinguish between these two types of comments. Additionally, many comments are irrelevant for both the product and the video (off-topic) or may even contain spam. Given that the main goal of sentiment analysis is to select sentiment-bearing comments and identify their polarity, distinguishing between off-topic and spam categories is not critical. Thus, we merge the spam and off-topic into a single uninformative category. Similar to the opinion classification task, comment type classification is a multi-class classification with three classes: video, product and uninform.

Full task. While the previously discussed sentiment and type identification tasks are useful to model and study in their own right, our end goal is: given a stream of comments, to jointly predict both the type and the sentiment of each comment. We cast this problem as a single multi-class classification task with seven classes: the Cartesian product between {product, video} type labels and {positive, neutral, negative} sentiment labels plus the uninformative category (spam and off-topic). Considering a real-life application, it is important not only to detect the polarity of the comment, but to also identify if it is expressed towards the product or the video.⁷⁷We exclude comments annotated as both video and product. This enables the use of a simple flat multi-classifiers with seven categories for the full task, instead of a hierarchical multi-label classifiers (i.e., type classification first and then opinion polarity). The number of comments assigned to both product and video is relatively small (8% for TABLETS and 4% for AUTO).

We split all the videos 50% between training set (TRAIN) and test set (TEST), where each video contains all its comments. This ensures that all comments from the same video appear either in TRAIN or in TEST. Since the number of comments per video varies, the resulting sizes of each set are different (we use the larger split for TRAIN). Table 1 shows the data distribution across the task-specific classes – sentiment and type classification. For the sentiment task we exclude off-topic and spam comments as well as comments with ambiguous sentiment, i.e., annotated as both positive and negative.

For the sentiment task about 50% of the comments have neutral polarity, while the negative class is much less frequent. Interestingly, the ratios between polarities expressed in comments from AUTO and TABLETS are very similar across both TRAIN and TEST. Conversely, for the type task, we observe that comments from AUTO are uniformly distributed among the three classes, while for the TABLETS the majority of comments are product related. It is likely due to the nature of the TABLETS videos, that are more geek-oriented, where users are more prone to share their opinions and enter involved discussions about a product. Additionally, videos from the AUTO category (both commercials and user reviews) are more visually captivating and, being generally oriented towards a larger audience, generate more video-related comments. Regarding the full setting, where the goal is to have a joint prediction of the comment sentiment and type, we observe that video-negative and video-positive are the most scarce classes, which makes them the most difficult to predict.

We start off by presenting the results for the traditional in-domain setting, where both TRAIN and TEST come from the same domain, e.g., AUTO or TABLETS. Next, we show the learning curves to analyze the behavior of FVEC and STRUCT models according to the training size. Finally, we perform a set of cross-domain experiments that describe the enhanced adaptability of the patterns generated by the STRUCT model.

Our STRUCT model is more accurate since it is able to induce structural patterns of sentiment. Consider the following comment: optimus pad is better. this xoom is just to bulky but optimus pad offers better functionality. The FVEC bag-of-words model misclassifies it to be positive, since it contains two positive expressions (better, better functionality) that outweigh a single negative expression (bulky). The structural model, in contrast, is able to identify the product of interest (xoom) and associate it with the negative expression through a structural feature and thus correctly classify the comment as negative.

Some issues remain problematic even for the structural model. The largest group of errors are implicit sentiments. Thus, some comments do not contain any explicit positive or negative opinions, but provide detailed and well-argumented criticism, for example, this phone is heavy. Such comments might also include irony. To account for these cases, a deep understanding of the product domain is necessary.