Most state-of-the-art wide-coverage parsers are trained on newspaper text and suffer a loss of accuracy in other domains , making parser adaptation a pressing issue . In this paper we demonstrate that a CCG parser can be adapted to two new domains , biomedical text and questions for a QA system , by using manually-annotated training data at the POS and lexical category levels only . This approach achieves parser accuracy comparable to that on newspaper data without the need for annotated parse trees in the new domain . We find that retraining at the lexical category level yields a larger performance increase for questions than for biomedical text and analyze the two datasets to investigate why different domains might behave differently for parser adaptation . Most state-of-the-art wide-coverage parsers are based on the Penn Treebank ( Marcus et al . , 1993 ) , making such parsers highly tuned to newspaper text . A pressing question facing the parsing community is how to adapt these parsers to other domains , such as biomedical research papers and web pages . A related question is how to improve the performance of these parsers on constructions that are rare in the Penn Treebank , such as questions . Questions are particularly important since a question parser is a component in most Question Answering ( QA ) systems ( Harabagiu et al . , 2001 ) . In this paper we investigate parser adaptation in the context of lexicalized grammars , by using a parser based on Combinatory Categorial Grammar ( CCG ) ( Steedman , 2000 ) . A key property of CCG is that it is lexicalized , meaning that each word in a sentence is associated with an elementary syntactic structure . In the case of CCG this is a lexical category expressing subcategorization information . We exploit this property of CCG by performing manual annotation in the new domain , but only up to this level of representation , where the annotation can be carried out relatively quickly . Since CCG lexical categories are so expressive , many of the syntactic characteristics of a domain are captured at this level . The two domains we consider are the biomedical domain and questions for a QA system . We use the term " domain " somewhat loosely here , since questions are best described as a particular set of syntactic constructions , rather than a set of documents about a particular topic . However , we consider question data to be interesting in the context of domain adaptation for the following reasons : 1 ) there are few examples in the Penn Treebank ( PTB ) and so PTB parsers typically perform poorly on them ; 2 ) questions form a fairly homogeneous set with respect to the syntactic constructions employed , and it is an interesting question how easy it is to adapt a parser to such data ; and 3 ) QA is becoming an important example of NLP technology , and question parsing is an important task for QA systems . The CCG parser we use ( Clark and Curran , 2007b ) makes use of three levels of representation : one , a POS tag level based on the fairly coarse-grained POS tags in the Penn Treebank ; two , a lexical category level based on the more fine-grained CCG lexical categories , which are assigned to words by a CCG su-pertagger ; and three , a hierarchical level consisting of CCG derivations . A key idea in this paper , following a pilot study in Clark et al . ( 2004 ) , is to perform manual annotation only at the first two levels . Since the lexical category level consists of sequences of tags , rather than hierarchical derivations , the annotation can be performed relatively quickly . For the biomedical and question domains we manually annotated approximately 1,000 and 2,000 sentences , respectively , with CCG lexical categories . We also created a gold standard set of grammatical relations ( GR ) in the Stanford format ( de Marneffe et al . , 2006 ) , using 500 of the questions . For the biomedical domain we used the BioInfer corpus ( Pyysalo et al . , 2007a ) , an existing gold-standard GR resource also in the Stanford format . We evaluated the parser on both lexical category assignment and recovery of GRs . The results show that the domain adaptation approach used here is successful in two very different domains , achieving parsing accuracy comparable to state-of-the-art accuracy for newspaper text . The results also show , however , that the two domains have different profiles with regard to the levels of representation used by the parser . We find that simply retraining the POS tagger used by the parser leads to a large improvement in performance for the biomedical domain , and that retraining the CCG supertagger on the annotated biomedical data improves the performance further . For the question data , retraining just the POS tagger also improves parser performance , but retraining the supertagger has a much greater effect . We perform some analysis of the two datasets in order to explain the different behaviours with regard to porting the CCG parser . We have targeted lower levels of representation in order to adapt a lexicalized-grammar parser to two new domains , biomedical text and questions . Although each of the lower levels has been targeted independently in previous work , this is the first study that examines both levels together to determine how they affect parsing accuracy . We achieved an accuracy on grammatical relations in the same range as that of the original parser for newspaper text , without requiring costly annotation of full parse trees . Both biomedical and question data are domains in which there is an immediate need for accurate parsing . The question dataset is in some ways an extreme example for domain adaptation , since the sentences are syntactically uniform ; on the other hand , it is of interest as a set of constructions where the parser initially performed poorly , and is a realistic parsing challenge in the context of QA systems . Interestingly , although an increase in accuracy at each stage of the pipeline did yield an increase at the following stage , these increases were not uniform across the two domains . The new POS tagger model was responsible for most of the improvement in parsing for the biomedical domain , while the new supertagger model was necessary to see a large improvement in the question domain . We attribute this to the fact that question syntax is significantly different from newspaper syntax . We expect these considerations to apply to any lexicalized-grammar parser . Of course , it would be useful to have a way of predicting which level of annotation would be most effective for adapting to a new domain before the annotation begins . The utility of measures such as unknown word rate ( which can be performed with unlabelled data ) and unknown POS n-gram rate ( which can be performed with only POS tags ) is not yet sufficiently clear to rely on them as predictive measures , but it seems a fruitful avenue for future work to investigate the importance of such measures for parser domain adaptation .
Most existing parsers are tuned for newspaper texts making them limited in applicable domains.
They propose a method to adapt a CCG parser to new domains using manually-annotated data only at POS and lexical category levels.
The proposed method achieves comparable results to in-domain parsers without expensive full annotations on biomedical texts and questions that are rare in existing benchmark datasets.