Semantic Frame Identification with Distributed Word Representations

Closely related to SRL, frame-semantic parsing consists of the resolution of predicate sense into a frame, and the analysis of the frame’s arguments. Work in this area exclusively uses the FrameNet full text annotations. Johansson and Nugues (2007) presented the best performing system at SemEval 2007 [1], and Das et al. (2010) improved performance, and later set the current state of the art on this task [7]. We briefly discuss FrameNet, and subsequently PropBank annotation conventions here.

FrameNet  The FrameNetproject [2] is a lexical database that contains information about words and phrases (represented as lemmas conjoined with a coarse part-of-speech tag) termed as lexical units, with a set of semantic frames that they could evoke. For each frame, there is a list of associated frame elements (or roles, henceforth), that are also distinguished as core or non-core.²²Additional information such as finer distinction of the coreness properties of roles, the relationship between frames, and that of roles are also present, but we do not leverage that information in this work. Sentences are annotated using this universal frame inventory. For example, consider the pair of sentences in Figure 1(a). Commerce_buy is a frame that can be evoked by morphological variants of the two example lexical units buy.V and sell.V. Buyer, Seller and Goods are some example roles for this frame.

PropBank  The PropBankproject [22] is another popular resource related to semantic role labeling. The PropBankcorpus has verbs annotated with sense frames and their arguments. Like FrameNet, it also has a lexical database that stores type information about verbs, in the form of sense frames and the possible semantic roles each frame could take. There are modifier roles that are shared across verb frames, somewhat similar to the non-core roles in FrameNet. Figure 1(b) shows annotations for two verbs “bought” and “sold”, with their lemmas (akin to the lexical units in FrameNet) and their verb frames buy.01 and sell.01. Generic core role labels (of which there are seven, namely A0-A5 and AA) for the verb frames are marked in the figure.³³NomBank [19] is a similar resource for nominal predicates, but we do not consider it in our experiments. A key difference between the two annotation systems is that PropBankuses a local frame inventory, where frames are predicate-specific. Moreover, role labels, although few in number, take specific meaning for each verb frame. Figure 1 highlights this difference: while both sell.v and buy.v are members of the same frame in FrameNet, they evoke different frames in PropBank. In spite of this difference, nearly identical statistical models could be employed for both frameworks.

Modeling  In this paper, we model the frame-semantic parsing problem in two stages: frame identification and argument identification. As mentioned in §1, these correspond to a frame disambiguation stage,⁴⁴For example in PropBank, the lexical unit buy.V has three verb frames and in sentential context, we want to disambiguate its frame. (Although PropBank never formally uses the term lexical unit, we adopt its usage from the frame semantics literature.) and a stage that finds the various arguments that fulfill the frame’s semantic roles within the sentence, respectively. This resembles the framework of Das et al. (2014), who solely focus on FrameNet corpora, unlike this paper. The novelty of this paper lies in the frame identification stage (§3). Note that this two-stage approach is unusual for the PropBank corpora when compared to prior work, where the vast majority of published papers have not focused on the verb frame disambiguation problem at all, only focusing on the role labeling stage (see the overview paper of Màrquez et al. (2008) for example).

We present a model that takes word embeddings as input and learns to identify semantic frames. A word embedding is a distributed representation of meaning where each word is represented as a vector in ${ℝ}^n$. Such representations allow a model to share meaning between similar words, and have been used to capture semantic, syntactic and morphological content [6, 25, inter alia]. We use word embeddings to represent the syntactic context of a particular predicate instance as a vector. For example, consider the sentence “He runs the company.” The predicate runs has two syntactic dependents – a subject and direct object (but no prepositional phrases or clausal complements). We could represent the syntactic context of runs as a vector with blocks for all the possible dependents warranted by a syntactic parser; for example, we could assume that positions $0\mathrm{\dots }n$ in the vector correspond to the subject dependent, $n+1\mathrm{\dots }2n$ correspond to the clausal complement dependent, and so forth. Thus, the context is a vector in ${ℝ}^{nk}$ with the embedding of He at the subject position, the embedding of company in direct object position and zeros everywhere else. Given input vectors of this form for our training data, we learn a matrix that maps this high dimensional and sparse representation into a lower dimensional space. Simultaneously, the model learns an embedding for all the possible labels (i.e. the frames in a given lexicon). At inference time, the predicate-context is mapped to the low dimensional space, and we choose the nearest frame label as our classification. We next describe this model in detail.

In principle $g\left(x\right)$ could be any feature function, but we performed an initial investigation of two particular variants. In both variants, our representation is a block vector where each block corresponds to a syntactic position relative to the predicate, and each block’s values correspond to the embedding of the word at that position.

Direct Dependents  The first context function we considered corresponds to the examples in §3. To elaborate, the positions of interest are the labels of the direct dependents of the predicate, so $k$ is the number of labels that the dependency parser can produce. For example, if the label on the edge between runs and He is nsubj, we would put the embedding of He in the block corresponding to nsubj. If a label occurs multiple times, then the embeddings of the words below this label are averaged.

Unfortunately, using only the direct dependents can miss a lot of useful information. For example, topicalization can place discriminating information farther from the predicate. Consider “He runs the company.” vs. “It was the company that he runs.” In the second sentence, the discriminating word, company dominates the predicate runs. Similarly, predicates in embedded clauses may have a distant agent which cannot be captured using direct dependents. Consider “The athlete ran the marathon.” vs. “The athlete prepared himself for three months to run the marathon.” In the second example, for the predicate run, the agent The athlete is not a direct dependent, but is connected via a longer dependency path.

Dependency Paths  To capture more relevant context, we developed a second context function as follows. We scanned the training data for a given task (either the PropBank or the FrameNet domains) for the dependency paths that connected the gold predicates to the gold semantic arguments. This set of dependency paths were deemed as possible positions in the initial vector space representation. In addition, akin to the first context function, we also added all dependency labels to the context set. Thus for this context function, the block cardinality $k$ was the sum of the number of scanned gold dependency path types and the number of dependency labels. Given a predicate in its sentential context, we therefore extract only those context words that appear in positions warranted by the above set. See Figure 3 for an illustration of this process.

We performed initial experiments using context extracted from 1) direct dependents, 2) dependency paths, and 3) both. For all our experiments, setting 3) which concatenates the direct dependents and dependency path always dominated the other two, so we only report results for this setting.

We model our objective function following Weston et al. (2011), using a weighted approximate-rank pairwise loss, learned with stochastic gradient descent. The mapping from $g\left(x\right)$ to the low dimensional space ${ℝ}^m$ is a linear transformation, so the model parameters to be learnt are the matrix $M\in {ℝ}^{nk\times m}$ as well as the embedding of each possible frame label, represented as another matrix $Y\in {ℝ}^{F\times m}$ where there are $F$ frames in total. The training objective function minimizes:

where $x,y$ are the training inputs and their corresponding correct frames, and $\overline{y}$ are negative frames, $\gamma$ is the margin. Here, $ran{k}_y\left(x\right)$ is the rank of the positive frame $y$ relative to all the negative frames:

and $L\left(\eta \right)$ converts the rank to a weight. Choosing $L\left(\eta \right)=C\eta$ for any positive constant $C$ optimizes the mean rank, whereas a weighting such as $L\left(\eta \right)={\sum }_{i=1}^{\eta }1/i$ (adopted here) optimizes the top of the ranked list, as described in [26]. To train with such an objective, stochastic gradient is employed. For speed the computation of $ran{k}_y\left(x\right)$ is then replaced with a sampled approximation: sample $N$ items $\overline{y}$ until a violation is found, i.e. $max(0,\gamma +s(x,\overline{y})-s(x,y)))>0$ and then approximate the rank with $\left(F-1\right)/N$, see Weston et al. (2011) for more details on this procedure. For the choices of the stochastic gradient learning rate, margin ($\gamma$) and dimensionality ($m$), please refer to §5.4-§5.5.

Note that an alternative approach could learn only the matrix $M$, and then use a $k$-nearest neighbor classifier in ${ℝ}^m$, as in Weinberger and Saul (2009). The advantage of learning an embedding for the frame labels is that at inference time we need to consider only the set of labels for classification rather than all training examples. Additionally, since we use a frame lexicon that gives us the possible frames for a given predicate, we usually only consider a handful of candidate labels. If we used all training examples for a given predicate for finding a nearest-neighbor match at inference time, we would have to consider many more candidates, making the process very slow.

We evaluate our models on both FrameNet- and PropBank-style structures. For FrameNet, we use the full-text annotations in the FrameNet 1.5 release⁸⁸See https://framenet.icsi.berkeley.edu. which was used by §3.2]das-etal-semafor-2013. We used the same test set as Das et al. containing 23 documents with 4,458 predicates. Of the remaining 55 documents, 16 documents were randomly chosen for development.⁹⁹These documents are listed in appendix A.

For experiments with PropBank, we used the Ontonotes corpus [14], version 4.0, and only made use of the Wall Street Journal documents; we used sections 2-21 for training, section 24 for development and section 23 for testing. This resembles the setup used by Punyakanok et al. (2008). All the verb frame files in Ontonotes were used for creating our frame lexicon.

For comparison, we implemented a set of baseline models, with varying feature configurations. The baselines use a log-linear model that models the following probability at training time:

At test time, this model chooses the best frame as ${\text{argmax}}_y𝝍\cdot 𝐟\left(y,x,\mathrm{\ell }\right)$ where argmax iterates over the possible frames $y\in F_{\mathrm{\ell }}$ if $\mathrm{\ell }$ was seen in the lexicon or the training data, or $y\in F$, if it was unseen, like the disambiguation scheme of §3. We train this model by maximizing $L_2$ regularized log-likelihood, using L-BFGS; the regularization constant was set to 0.1 in all experiments.

For comparison with our model from §3, which we call Wsabie Embedding, we implemented two baselines with the log-linear model. Both the baselines use features very similar to the input representations described in §3.1. The first one computes the direct dependents and dependency paths as described in §3.1 but conjoins them with the word identity rather than a word embedding. Additionally, this model uses the un-conjoined words as backoff features. This would be a standard NLP approach for the frame identification problem, but is surprisingly competitive with the state of the art. We call this baseline Log-Linear Words. The second baseline, tries to decouple the Wsabietraining from the embedding input, and trains a log linear model using the embeddings. So the second baseline has the same input representation as Wsabie Embedding but uses a log-linear model instead of Wsabie. We call this model Log-Linear Embedding.

We process our PropBank and FrameNet training, development and test corpora with a shift-reduce dependency parser that uses the Stanford conventions [9] and uses an arc-eager transition system with beam size of 8; the parser and its features are described by Zhang and Nivre (2011). Before parsing the data, it is tagged with a POS tagger trained with a conditional random field [17] with the following emission features: word, the word cluster, word suffixes of length 1, 2 and 3, capitalization, whether it has a hyphen, digit and punctuation. Beyond the bias transition feature, we have two cluster features for the left and right words in the transition. We use Brown clusters learned using the algorithm of Uszkoreit and Brants (2008) on a large English newswire corpus for cluster features. We use the same word clusters for the argument identification features in Table 1.

We learn the initial embedding representations for our frame identification model (§3) using a deep neural language model similar to the one proposed by Bengio et al. (2003). We use 3 hidden layers each with 1024 neurons and learn a 128-dimensional embedding from a large corpus containing over 100 billion tokens. In order to speed up learning, we use an unnormalized output layer and a hinge-loss objective. The objective tries to ensure that the correct word scores higher than a random incorrect word, and we train with minibatch stochastic gradient descent.

Hyperparameters For our frame identification model with embeddings, we search for the Wsabiehyperparameters using the development data. We search for the stochastic gradient learning rate in $\left\{\underset{¯}{0.0001},0.001,0.01\right\}$, the margin $\gamma \in \left\{0.001,\underset{¯}{0.01},0.1,1\right\}$ and the dimensionality of the final vector space $m\in \left\{\underset{¯}{256},512\right\}$, to maximize the frame identification accuracy of ambiguous lexical units; by ambiguous, we imply lexical units that appear in the training data or the lexicon with more than one semantic frame. The underlined values are the chosen hyperparameters used to analyze the test data.

Argument Candidates The candidate argument extraction method used for the FrameNet data, (as mentioned in §4) was adapted from the algorithm of Xue and Palmer (2004) applied to dependency trees. Since the original algorithm was designed for verbs, we added a few extra rules to handle non-verbal predicates: we added 1) the predicate itself as a candidate argument, 2) the span ranging from the sentence position to the right of the predicate to the rightmost index of the subtree headed by the predicate’s head; this helped capture cases like “a few months” (where few is the predicate and months is the argument), and 3) the span ranging from the leftmost index of the subtree headed by the predicate’s head to the position immediately before the predicate, for cases like “your gift to Goodwill” (where to is the predicate and your gift is the argument).¹⁰¹⁰ Note that Das et al. (2014) describe the state of the art in FrameNet-based analysis, but their argument identification strategy considered all possible dependency subtrees in a parse, resulting in a much larger search space.

Frame Lexicon  In our experimental setup, we scanned the XML files in the “frames” directory of the FrameNet 1.5 release, which lists all the frames, the corresponding roles and the associated lexical units, and created a frame lexicon to be used in our frame and argument identification models. We noted that this renders every lexical unit as seen; in other words, at frame disambiguation time on our test set, for all instances, we only had to score the frames in $F_{\mathrm{\ell }}$ for a predicate with lexical unit $\mathrm{\ell }$ (see §3 and §5.2). We call this setup Full Lexicon. While comparing with prior state of the art on the same corpus, we noted that Das et al. (2014) found several unseen predicates at test time.¹¹¹¹ Instead of using the frame files, Das et al. built a frame lexicon from FrameNet’s exemplars and the training corpus. For fair comparison, we took the lexical units for the predicates that Das et al. considered as seen, and constructed a lexicon with only those; training instances, if any, for the unseen predicates under Das et al.’s setup were thrown out as well. We call this setup Semafor Lexicon.¹²¹²We got Das et al.’s seen predicates from the authors. We also experimented on the set of unseen instances used by Das et al.

ILP constraints  For FrameNet, we used three ILP constraints during argument identification (§4). 1) each span could have only one role, 2) each core role could be present only once, and 3) all overt arguments had to be non-overlapping.

Hyperparameters  As in §5.4, we made a hyperparameter sweep in the same space. The chosen learning rate was $0.01$, while the other values were $\gamma =0.01$ and $m=512$. Ambiguous lexical units were used for this selection process.

Argument Candidates  For PropBankwe use the algorithm of Xue and Palmer (2004) applied to dependency trees.

Frame Lexicon  For the PropBankexperiments we scanned the frame files for propositions in Ontonotes 4.0, and stored possible core roles for each verb frame. The lexical units were simply the verb associating with the verb frames. There were no unseen verbs at test time.

ILP constraints  We used the constraints of Punyakanok et al. (2008).

Table 2 presents accuracy results on frame identification.¹³¹³We do not report partial frame accuracy that has been reported by prior work. We present results on all predicates, ambiguous predicates seen in the lexicon or the training data, and rare ambiguous predicates that appear $\le 11$ times in the training data. The Wsabie Embedding model from §3 performs significantly better than the Log-Linear Words baseline, while Log-Linear Embedding underperforms in every metric. For the Semafor Lexicon setup, we also compare with the state of the art from Das et al. (2014), who used a semi-supervised learning method to improve upon a supervised latent-variable log-linear model. For unseen predicates from the Das et al. system, we perform better as well. Finally, for the Full Lexicon setting, the absolute accuracy numbers are even better for our best model. Table 3 presents results on the full frame-semantic parsing task (measured by a reimplementation of the SemEval 2007 shared task evaluation script) when our argument identification model (§4) is used after frame identification. We notice similar trends as in Table 2, and our results outperform the previously published best results, setting a new state of the art.

Table 4 shows frame identification results on the PropBankdata. On the development set, our best model performs with the highest accuracy on all and ambiguous predicates, but performs worse on rare ambiguous predicates. On the test set, the Log-Linear Words baseline performs best by a very narrow margin. See §6 for a discussion.

Table 5 presents results where we measure precision, recall and $F_1$ for frames and arguments together; this strict metric penalizes arguments for mismatched frames, like in Table 3. We see the same trend as in Table 4. Finally, Table 6 presents SRL results that measures argument performance only, irrespective of the frame; we use the evaluation script from CoNLL 2005 [5]. We note that with a better frame identification model, our performance on SRL improves in general. Here, too, the embedding model barely misses the performance of the best baseline, but we are at par and sometimes better than the single parser setting of a state-of-the-art SRL system [23].¹⁴¹⁴The last row of Table 6 refers to a system which used the combination of two syntactic parsers as input.