1. A method, in a data processing system comprising a processor and a memory, for dynamically selecting a subset of candidate answer sources for use by a question and answer (QA) system implemented by the data processing system, the method comprising: receiving, by the QA system implemented by the data processing system, an input question for which an answer is sought; generating, by the QA system, one or more queries based on the input question; selecting, by the QA system, a subset of candidate answer sources, from a group of candidate answer sources, based on a ranked listing of candidate answer sources, wherein the ranked listing of candidate answer sources is generated based on characteristics of previous candidate answers generated by the QA system for previously input questions; applying, by the QA system, the one or more queries to the subset of candidate answer sources; generating, by the QA system, at least one candidate answer for the input question based on results of applying the one or more queries to the subset of candidate answer sources; logging, for the previously input questions, in entries of at least one log data structure, the characteristics of previous candidate answers generated by the QA system, wherein the characteristics of each previous candidate answer comprises an identification of a candidate answer source from which the candidate answer was generated and a confidence score associated with the candidate answer; and generating the ranked listing of candidate answer sources based on the entries in the at least one log data structure.