The Penn Parsed Corpus of Modern British English:
First Parsing Results and Analysis

We present the first parsing results for the Penn Parsed Corpus of Modern British English (PPCMBE) [], showing that it can be parsed at a few points lower in F-score than the Penn Treebank (PTB) []. We discuss some of the differences in annotation style and source material that make a direct comparison problematic. Some first steps at analysis of the parsing results indicate aspects of the annotation style that are difficult for the parser, and also show that the parser is creating structures that are not present in the training material.

The PPCMBE is a million-word treebank created for researching changes in English syntax. It covers the years 1700-1914 and is the most modern in the series of treebanks created for historical research.¹¹The other treebanks in the series cover Early Modern English [] (1.8 million words), Middle Eng-lish [] (1.2 million words), and Early English Correspondence [] (2.2 million words). Due to the historical nature of the PPCMBE, it shares some of the characteristics of treebanks based on modern unedited text [], such as spelling variation.

The size of the PPCMBE is roughly the same as the WSJ section of the PTB, and its annotation style is similar to that of the PTB, but with differences, particularly with regard to coordination and NP structure. However, except for , we have found no discussion of this corpus in the literature.²² report some results on POS tagging using their own mapping to different tags, but no parsing results. There is also much additional material annotated in this style, increasing the importance of analyzing parser performance on this annotation style.³³Aside from the corpora listed in fn. 1, there are also historical corpora of Old English [], Icelandic [], French [], and Portuguese [], totaling 4.5 million words.

The PPCMBE⁴⁴We are working with a pre-release copy of the next revision of the official version. Some annotation errors in the currently available version have been corrected, but the differences are relatively minor. consists of 101 files, but we leave aside 7 files that consist of legal material with very different properties than the rest of the corpus. The remaining 94 files contain 1,018,736 tokens (words).

We refer to the pre-release version of the corpus described in Section 2 as the “Release” version, and experiment with three other corpus versions.

We split the data into four sections, as shown in Table 2. The validation section consists of the four files beginning with “a” or “v” (spanning the years 1711-1860), the development section consists of the four files beginning with “l” (1753-1866), the test section consists of the five files beginning with “f” (1749-1900), and the training section consists of the remaining 81 files (1712-1913). The data split sizes used here for the PPCMBE closely approximate that used for the PTB, as described in .⁷⁷Sections 2-21 for Training Section 1 for Val, 22 for Dev and 23 for Test. For this first work, we used a split that was roughly the same as far as time-spans across the four sections. In future work, we will do a more proper cross-validation evaluation.

Table 3 shows the average sentence length and percentage of sentences of length in the PPCMBE and PTB. The PPCMBE sentences are a bit longer on average, and fewer are of length . However, the match is close enough that we will report the parsing results for sentences of length and all sentences, as with the PTB.

The PPCMBE is a phrase-structure corpus, and so we parse with the Berkeley parser [] and score using the standard evalb program []. We used the Train and Val sections for training, with the parser using the Val section for fine-tuning parameters []. Since the Berkeley parser is capable of doing its own POS tagging, we ran it using the gold tags or supplying its own tags. Table 4 shows the results for both modes.⁸⁸We modified the evalb parameter file to exclude punctuation in PPCMBE, just as for PTB. The results are based on a single run for each corpus/section. We expect some variance to occur, and in future work will average results over several runs of the training/Dev cycle, following .

Consider first the results for the Dev section with the parser using the gold tags. The score for all sentences increases from 83.7 for the Release corpus (row 1) to 84.7 for the Reduced corpus (row 2), reflecting the POS tag simplifications in the Reduced corpus. The score goes up by a further 2.0 to 86.7 (row 2 to 4) for the Reduced$+$NPs corpus and up again by 0.4 to 87.1 (row 5) for the Reduced$+$NPs$+$VPs corpus, showing the effects of the extra NP and VP brackets. We evaluated the Test section on the Reduced corpus (row 3), with a result 0.8 higher than the Dev (85.5 in row 3 compared to 84.7 in row 2). The score for sentences of length (a larger percentage of the PPCMBE than the PTB) is 2.4 higher than the score for all sentences, with both the gold and parser tags (row 5).

The results with the parser choosing its own POS tags naturally go down, with the Test section suffering more. In general, the PPCMBE is affected by the lack of gold tags more than the PTB.

In sum, the parser results show that the PPCMBE can be parsed at a level approaching that of the PTB. We are not proposing that the current version be replaced by the Reduced$+$NPs$+$VPs version, on the grounds that the latter gets the highest score. Our goal was to determine whether the parsing results fell in the same general range as for the PTB by roughly compensating for the difference in annotation style. The results in Table 4 show that this is the case.

As a final note, the PPCMBE consists of unedited data spanning more than 200 years, while the PTB is edited newswire, and so to some extent there would almost certainly be some difference in score.

We are currently developing techniques to better understand the types of errors is making, which have already led to interesting results. The parser is creating some odd structures that violate basic well-formedness conditions of clauses. Tree (6a) in Figure 6 is a tree from from the “Reduced” corpus, in which the verb “formed” projects to IP, with two auxiliary verbs (“had” and “been”). In the corresponding parser output (6b), the parser misses the reduced relative RRC, turning “acting” into the rightmost verb in the IP. The parser is creating an IP with two main verbs - an ungrammatical structure that is not attested in the gold.

It might be thought that the parser is having trouble with the flat-IP annotation style, but the parser posits incorrect structures that are not attested in the gold even in the Reduced$+$NPs$+$VPs version of the corpus. Tree (6a) shows a fragment of a gold tree from the corpus, with the VPs appropriately inserted. The parser output (6b) has an extra IP above “teaching”. The POS tags for “be” (BE) and “teaching“ (VAG) do not appear in this configuration at all in the training material. In general, the parser seems to be getting confused as to when such an IP should appear. We hypothesized that this is due to confusion with infinitival clauses, which can have an unary-branching IP over a VP, as in the gold tree (6). We retrained the parser, directing it to retain the INF function tag that appears in infinitival clauses as in (6). Overall, the evalb score went down slightly, but it did fix cases such as (6b). We do not yet know why the overall score went down, but what’s surprising is one would have thought that IP-INF is recoverable from the absence of a tensed verb.

Preliminary analysis shows that the CONJP structures are also difficult for the parser. Since these are structures that are different than the PTB⁹⁹The CONJP nonterminal in the PTB serves a different purpose than in the PPCMBE and is much more limited., we were particularly interested in them. Cases where the CONJP is missing an overt coordinating cord (such as “and”), are particularly difficult, not surprisingly. These can appear as intermediate conjuncts in a string of conjuncts, with the structure (CONJP word). The shared pre-modifier structure described in (2a) is also difficult for the parser.

We have presented the first results on parsing the PPCMBE and discussed some significant annotation style differences from the PTB. Adjusting for two major differences that are a matter of annotation convention, we showed that the PPCMBE can be parsed at approximately the same level of accuracy as the PTB. The first steps in an investigation of the parser differences show that the parser is generating structures that violate basic well-formedness conditions of the annotation.

For future work, we will carry out a more serious analysis of the parser output, trying to more properly account for the differences in bracketing structure between the PPCMBE and PTB. There is also a great deal of data annotated in the style of the PPCMBE, as indicated in footnotes 1 and 1, and we are interested in how the parser performs on these, especially comparing the results on the modern English corpora to the older historical ones, which will have greater issues of orthographic and tokenization complications.

This work was supported by National Science Foundation Grant # BCS-114749. We would like to thank Ann Bies, Justin Mott, and Mark Liberman for helpful discussions.