This paper introduces a new parser evaluation corpus containing around 700 sentences annotated with unbounded dependencies , from seven different grammatical constructions . We run a series of off-theshelf parsers on the corpus to evaluate how well state-of-the-art parsing technology is able to recover such dependencies . The overall results range from 25 % accuracy to 59 % . These low scores call into question the validity of using Parseval scores as a general measure of parsing capability . We discuss the importance of parsers being able to recover unbounded dependencies , given their relatively low frequency in corpora . We also analyse the various errors made on these constructions by one of the more successful parsers . Statistical parsers are now obtaining Parseval scores of over 90 % on the WSJ section of the Penn Treebank ( Bod , 2003 ; Petrov and Klein , 2007 ; Huang , 2008 ; Carreras et al . , 2008 ) . McClosky et al . ( 2006 ) report an F-score of 92.1 % using selftraining applied to the reranker of Charniak and Johnson ( 2005 ) . Such scores , in isolation , may suggest that statistical parsing is close to becoming a solved problem , and that further incremental improvements will lead to parsers becoming as accurate as POS taggers . A single score in isolation can be misleading , however , for a number of reasons . First , the single score is an aggregate over a highly skewed distribution of all constituent types ; evaluations which look at individual constituent or dependency types show that the accuracies on some , semantically important , constructions , such as coordination and PP-attachment , are much lower ( Collins , 1999 ) . Second , it is well known that the accuracy of parsers trained on the Penn Treebank degrades when they are applied to different genres and domains ( Gildea , 2001 ) . Finally , some researchers have argued that the Parseval metrics ( Black et al . , 1991 ) are too forgiving with respect to certain errors and that an evaluation based on syntactic dependencies , for which scores are typically lower , is a better test of parser performance ( Lin , 1995 ; Carroll et al . , 1998 ) . In this paper we focus on the first issue , that the performance of parsers on some constructions is much lower than the overall score . The constructions that we focus on are various unbounded dependency constructions . These are interesting for parser evaluation for the following reasons : one , they provide a strong test of the parser 's knowledge of the grammar of the language , since many instances of unbounded dependencies are difficult to recover using shallow techniques in which the grammar is only superficially represented ; and two , recovering these dependencies is necessary to completely represent the underlying predicateargument structure of a sentence , useful for applications such as Question Answering and Information Extraction . To give an example of the sorts of constructions we are considering , and the ( in)ability of parsers to recover the corresponding unbounded dependencies , none of the parsers that we have tested were able to recover the dependencies shown in bold from the following sentences : We have also developed techniques for recognizing and locating underground nuclear tests through the waves in the ground which they generate . By Monday , they hope to have a sheaf of documents both sides can trust . In this paper we have demonstrated that current parsing technology is poor at recovering some of the unbounded dependencies which are crucial for fully representing the underlying predicateargument structure of a sentence . We have also argued that correct recovery of such dependencies will become more important as parsing technology improves , despite the relatively low frequency of occurrence of the corresponding grammatical constructions . We also see this more focused parser evaluation methodology -in this case construction-focused -as a way of improving parsing technology , as an alternative to the exclusive focus on incremental improvements in overall accuracy measures such as Parseval .
While recent statistical parsers perform well on Penn Treebank, the results can be misleading due to several reasons originating from evaluation and datasets.
They propose a new corpus with unbounded dependencies from difference grammatical constructions.
Their evaluation of existing parsers with the proposed corpus shows lower scores than reported in previous works indicating a poor ability to recover unbounded dependencies.