Toward Future Scenario Generation: Extracting Event Causality Exploiting Semantic Relation, Context, and Association Features

We hypothesize that two nouns with some particular semantic relations are more likely to constitute event causality. Below we describe the semantic relations that we believe are likely to constitute event causality.

Causation is the causal relation between two entities and is expressed by binary patterns like A causes B. Deforestation and global warming might complete the A and B slots. We manually collected 748 binary patterns for this relation. (See Section LABEL:A-sec:examples-of-binary-patterns in the supplementary notes for examples of our binary patterns.)

Material is the relation between a material and a product made of it (e.g. plutonium and atomic bomb) and can be expressed by A is made of B. Its relation to event causality might seem unclear, but a material can be seen as a “cause” of a product. Indeed materials can participate in event causality with the help of such template pairs as A is stolen$\mathrm{\to }$B is made as in plutonium is stolen$\mathrm{\to }$atomic bomb is made. We manually collected 187 binary patterns for this relation.

Necessity’s patterns include A is necessary for B, which can be filled with verbal aptitude and ability to think. Noun pairs with this relation can constitute event causality when combined with template pairs like improve A$\mathrm{\to }$cultivate B. We collected 257 patterns for this relation.

Use is the relation between means (or instruments) and the purpose for using them. A is used for B is a pattern of the relation, which can be filled with e-mailer and exchanges of e-mail messages. Note that means can be seen as “causing” or “realizing” the purpose of using the means in this relation, and actually event causality can be obtained by incorporating noun pairs of this relation into template pairs like activate A$\mathrm{\to }$conduct B. 2,178 patterns were collected for this relation.

Prevention is the relation expressed by patterns like A prevents B, which can be filled with toothbrushing and periodontal disease. This relation is, so to speak, “negative Causation” since the entity denoted by the noun completing the A slot makes the entity denoted by the B noun NOT realized. Such noun pairs mean event causality by substituting them into template pairs like omit A$\mathrm{\to }$get B. The number of patterns is 490.

The experiments in Section 5.1.1 show that not only Causation and Prevention (“negative Causation”) but the other relations are also effective for event causality extraction.

In addition, we invented the \UnderlineExcitation relation that is expressed by binary patterns made of excitatory and inhibitory templates (Section 3.1). For instance, we make binary patterns A rises B and A lowers B from excitatory template rise X and inhibitory template lower X respectively. The Excitation relation roughly means that A activates B (excitatory) or suppresses it (inhibitory). We simply add an additional argument position to each template in the 43,697 excitation templates to make binary patterns. We restricted the argument positions (represented by Japanese postpositions) of the A slot to either ha (topic marker), ga (nominative), or de (instrumental) and those of the B slot to either ha, ga, de, wo (accusative), or ni (dative), and obtained 55,881 patterns.

Moreover, for broader coverage, we acquired binary patterns that entail or are entailed by one of the patterns of the above six semantic relations. Those patterns were acquired from our web corpus by Kloetzer et al. (2013b)’s method, which acquired 185 million entailment pairs with 80% precision from our web corpus and was used for contradiction acquisition [14]. We acquired 335,837 patterns by this method. They are class-dependent patterns, which have semantic class restrictions on arguments. The semantic classes were obtained from our web corpus based on Kazama and Torisawa (2008). See De Saeger et al. (2009), De Saeger et al. (2011) and Kloetzer et al. (2013a) for more on our patterns. They collectively constitute the \UnderlineEntailment relation.

Table 1

shows our semantic relation features. To use them, we first make a database that records which noun pairs co-occur with each binary pattern. Then we check a noun pair (the nouns of the cause and effect phrases) for each event causality candidate, and give the candidate all the patterns in the database that co-occur with the noun pair.

We believe that contexts exist where event causality candidates are more likely to appear, as described in Section 1. We developed features that capture the characteristics of likely contexts for Japanese event causality (See Section LABEL:A-sec:context-features in the supplementary notes). In a nutshell, they represent a connective (C1 and C2 in Section LABEL:A-sec:context-features), the distance between the elements of event causality candidate (C3 and C4), words in context (C5 to C8), the existence of adnominal modifier (9 to C10), and the existence of additional arguments of cause and effect predicates (C13 to C20), among others.

These features measure the association strength between slash-and-burn agriculture and desertification in conduct slash-and-burn agriculture$\mathrm{\to }$exacerbate desertification for instance and consist of CEA-, Wikipedia-, definition-, and web-based features. CEA-based features are based on the Cause Effect Association (CEA) measure of Do et al. (2011). It consists of association measures like PMI between arguments (nouns), between arguments and predicates, and between predicates (Table 2). Do et al. used it (along with discourse relations) to extract event causality.

Wikipedia-based features are the co-occurrence counts and the PMI values between cause and effect nouns calculated using Wikipedia (as of 2013-Sep-19). We also checked whether an Wikipedia article whose title is a cause (effect) noun contains its effect (cause) noun, as detailed in Section LABEL:A-sec:wikipedia-based-association-features in the supplementary notes. Definition-based features, as detailed in Section LABEL:A-sec:definition-based-association-features in the supplementary notes, resemble the Wikipedia-based features except that the information source is the definition sentences automatically acquired from our 600 million web pages using the method of Hashimoto et al. (2011). Web-based features provide association measures between nouns using various window sizes in the 600 million web pages. See Section LABEL:A-sec:web-based-association-features for detail. Web-based association measures were obtained from the same database as AC4 in Table 2.

Base features represent the basic properties of event causality like nouns, templates, and their excitation polarities (See Section LABEL:A-sec:base-features in the supplementary notes). For B3 and B4, 500 semantic classes were obtained from our web corpus using the method of Kazama and Torisawa (2008).

We evaluated the features of our method by ablation tests. Table 3 shows the results of removing the semantic relation, the context, and the association features from our method.

All the feature types are effective and contribute to the performance gain that was about 5% higher than the Base features only. Proposed achieved 70% precision at 13% recall. We then estimated that, with the precision rate, we can extract 69,700 event causality pairs from the 2,451,254 event causality candidates, among which the estimated number of positive examples is 377,794.

Next we examined whether the semantic relations that do not seem directly relevant to causality like Material are effective. Table 4 shows that

the performance degraded (46.27 $\to$ 45.86) when we only used the Causation binary patterns and their entailing and entailed patterns compared to Proposed. Even when adding the Prevention (“negative Causation”) patterns and their entailing and entailed patterns, the performance was still slightly worse than Proposed. The performance was even worse when using no semantic relation (“None” in Table 4). Consequently we conclude that not only semantic relations directly relevant to causality like Causation but also those that seem to lack direct relevance to causality like Material are somewhat effective.

Finally, Table 5 shows the performance drop by removing the Wikipedia-, definition-, web-, and CEA-based features.

The CEA-based features were the most effective, while the Wikipedia-based ones slightly degraded the performance.

We compared our method and two baselines based on Do et al. (2011): CEA${}_{u\mathbf{}n\mathbf{}s}$ is an unsupervised method that uses CEA to rank event causality candidates, and CEA${}_{s\mathbf{}u\mathbf{}p}$ is a supervised method using SVM and the CEA features, whose ranking is based on the SVM scores. The baselines are not complete implementations of Do et al.’s method which uses discourse relations identified based on Lin et al. (2010) and exploits them with CEA within an ILP framework. Nonetheless, we believe that this comparison is informative since CEA can be seen as the main component; they achieved a F1 of 41.7% for extracting causal event relations, but with only CEA they still achieved 38.6%.

Table 6 shows the average precision of the compared methods.

Proposed is our proposed method. Proposed-CEA is Proposed without the CEA-features and shows their contribution. Proposed is the best and the CEA features slightly contribute to the performance, as Proposed-CEA indicates. We observed that CEA${}_{s\mathbf{}u\mathbf{}p}$ and CEA${}_{u\mathbf{}n\mathbf{}s}$ performed poorly and tended to favor event causality candidates whose phrase pairs were highly relevant to each other but described the contrasts of events rather than event causality (e.g. build a slow muscle and build a fast muscle) probably because their main components are PMI values. Figure 1 shows their precision-recall curves.

Next we compared our method with the baselines based on Hashimoto et al. (2012). They developed an automatic excitation template acquisition method that assigns each template an excitation value in range $\left[-1,1\right]$ that is positive if the template is excitatory and negative if it is inhibitory. They ranked event causality candidates by $Cs\left(p_1,p_2\right)=\left|s_1\right|\times \left|s_2\right|$, where $p_1$ and $p_2$ are the two phrases of event causality candidates, and $\left|s_1\right|$ and $\left|s_2\right|$ are the absolute excitation values of $p_1$’s and $p_2$’s templates. The baselines are as follows: Cs${}_{u\mathbf{}n\mathbf{}s}$ is an unsupervised method that uses $Cs$ for ranking, and Cs${}_{s\mathbf{}u\mathbf{}p}$ is a supervised method using SVM with $Cs$ as the only feature that uses SVM scores for ranking. Note that some event causality candidates were not given excitation values for their templates, since some templates were acquired by manual annotation without Hashimoto et al.’s method. To favor the baselines for fairness, the event causality candidates of the development and test data were restricted to those with excitation values. Since Cs${}_{s\mathbf{}u\mathbf{}p}$ performed slightly better when using all of the training data in our preliminary experiments, we used all of it.

Table 7 shows the average precision of the compared methods.

Proposed is our method. Its average precision is different from that in Table 6 due to the difference in test data described above. Cs${}_{u\mathbf{}n\mathbf{}s}$ and Cs${}_{s\mathbf{}u\mathbf{}p}$ did not perform well. Many phrase pairs described two events that often happen in parallel but are not event causality (e.g. reduce the intake of energy and increase the energy consumption) in the highly ranked event causality candidates of Cs${}_{u\mathbf{}n\mathbf{}s}$ and Cs${}_{s\mathbf{}u\mathbf{}p}$. Figure 2 shows their precision-recall curves.

Hashimoto et al. (2012) extracted 500,000 event causalities with about 70% precision. However, as described in Section 1, our event causality criteria are different; since they regarded phrase pairs that were not self-contained as event causality (their annotators checked the original sentences of phrase pairs to see if they were event causality), their judgments tended to be more lenient than ours, which explains the performance difference.

In preliminary experiments, since our proposed method’s performance degraded when $Cs$ was incorporated, we did not use it in our method.