1. A method, in a question and answer (QA) system comprising a processor and a memory, for retrieving candidate answers from a corpus of documents, the method comprising: receiving, by the QA system, an input question for which an answer is sought; extracting, by the QA system, features of the input question based on a natural language processing of the input question; executing, by the QA system, a first search of the corpus of documents based on a first subset of the extracted features of the input question and an initial evaluation of a utility of the first subset of extracted features to generate a subset of documents matching the first subset of extracted features; executing, by the QA system, a second search of a set of passages extracted from the subset of documents based on a second subset of the extracted features of the input question and a reevaluation of the utility of the second subset of extracted features thereby forming a subset of passages; and generating, by the QA system, query results from the subset of passages from which a set of candidate answers for the input question are identified.