In this paper , we explore correlation of dependency relation paths to rank candidate answers in answer extraction . Using the correlation measure , we compare dependency relations of a candidate answer and mapped question phrases in sentence with the corresponding relations in question . Different from previous studies , we propose an approximate phrase mapping algorithm and incorporate the mapping score into the correlation measure . The correlations are further incorporated into a Maximum Entropy-based ranking model which estimates path weights from training . Experimental results show that our method significantly outperforms state-ofthe-art syntactic relation-based methods by up to 20 % in MRR . Answer Extraction is one of basic modules in open domain Question Answering ( QA ) . It is to further process relevant sentences extracted with Passage / Sentence Retrieval and pinpoint exact answers using more linguistic-motivated analysis . Since QA turns to find exact answers rather than text snippets in recent years , answer extraction becomes more and more crucial . Typically , answer extraction works in the following steps : • Recognize expected answer type of a question . • Annotate relevant sentences with various types of named entities . • Regard the phrases annotated with the expected answer type as candidate answers . • Rank candidate answers . In the above work flow , answer extraction heavily relies on named entity recognition ( NER ) . On one hand , NER reduces the number of candidate answers and eases answer ranking . On the other hand , the errors from NER directly degrade answer extraction performance . To our knowledge , most top ranked QA systems in TREC are supported by effective NER modules which may identify and classify more than 20 types of named entities ( NE ) , such as abbreviation , music , movie , etc . However , developing such named entity recognizer is not trivial . Up to now , we have n't found any paper relevant to QA-specific NER development . So , it is hard to follow their work . In this paper , we just use a general MUC-based NER , which makes our results reproducible . A general MUC-based NER ca n't annotate a large number of NE classes . In this case , all noun phrases in sentences are regarded as candidate answers , which makes candidate answer sets much larger than those filtered by a well developed NER . The larger candidate answer sets result in the more difficult answer extraction . Previous methods working on surface word level , such as density-based ranking and pattern matching , may not perform well . Deeper linguistic analysis has to be conducted . This paper proposes a statistical method which exploring correlation of dependency relation paths to rank candidate answers . It is motivated by the observation that relations between proper answers and question phrases in candidate sentences are always similar to the corresponding relations in question . For example , the question " What did Alfred Nobel invent ? " and the candidate sentence " ... in the will of Swedish industrialist Alfred Nobel , who invented dynamite . " For each question , firstly , dependency relation paths are defined and extracted from the question and each of its candidate sentences . Secondly , the paths from the question and the candidate sentence are paired according to question phrase mapping score . Thirdly , correlation between two paths of each pair is calculated by employing Dynamic Time Warping algorithm . The input of the calculation is correlations between dependency relations , which are estimated from a set of training path pairs . Lastly , a Maximum Entropy-based ranking model is proposed to incorporate the path correlations and rank candidate answers . Furthermore , sentence supportive measure are presented according to correlations of relation paths among question phrases . It is applied to re-rank the candidate answers extracted from the different candidate sentences . Considering phrases may provide more accurate information than individual words , we extract dependency relations on phrase level instead of word level . The experiment on TREC questions shows that our method significantly outperforms a densitybased method by 50 % in MRR and three stateof-the-art syntactic-based methods by up to 20 % in MRR . Furthermore , we classify questions by judging whether NER is used . We investigate how these methods perform on the two question sets . The results indicate that our method achieves better performance than the other syntactic-based methods on both question sets . Especially for more difficult questions , for which NER may not help , our method improves MRR by up to 31 % . The paper is organized as follows . Section 2 discusses related work and clarifies what is new in this paper . Section 3 presents relation path correlation in detail . Section 4 and 5 discuss how to incorporate the correlations for answer ranking and re-ranking . Section 6 reports experiment and results . In this paper , we propose a relation path correlation-based method to rank candidate answers in answer extraction . We extract and pair relation paths from questions and candidate sentences . Next , we measure the relation path correlation in each pair based on approximate phrase mapping score and relation sequence alignment , which is calculated by DTW algorithm . Lastly , a ME-based ranking model is proposed to incorporate the path correlations and rank candidate answers . The experiment on TREC questions shows that our method significantly outperforms a density-based method by 50 % in MRR and three state-of-the-art syntactic-based methods by up to 20 % in MRR . Furthermore , the method is especially effective for difficult questions , for which NER may not help . Therefore , it may be used to further enhance state-of-the-art QA systems even if they have a good NER . In the future , we are to further evaluate the method based on the overall performance of a QA system and adapt it to sentence retrieval task .
A generally accessible NER system for QA systems produces a larger answer candidate set which would be hard for current surface word-level ranking methods.
They propose a statistical method which takes correlations of dependency relation paths computed by the Dynamic Time Wrapping algorithm into account for ranking candidate answers.
The proposed method outperforms state-of-the-art syntactic relation-based methods by up to 20% and shows it works even better on harder questions where NER performs poorly.