1. A method, in an information handling system comprising a processor and a memory, of improving a quality of question-answer sets used as inputs to a question-answering (QA) system, the method comprising: receiving, from a user interface that is utilized by a user, a question-answer pair; analyzing the question-answer pair using natural language processing (NLP) components, wherein the question-answer pair includes a question and an answer to the question, and wherein at least one of the NLP components is a configurable analyzer extension; identifying, based on the analysis, one or more shortcomings of the question-answer pair, wherein the shortcomings relate to an ability of the target QA system to analyze the question; providing a human-readable feedback to the user by transmitting the human-readable feedback to the user interface, wherein the feedback recommends one or more possible actions to address the identified shortcomings; and receiving, from the user interface, a modified question-answer pair input by the user, wherein the modified question-answer pair addresses at least one of the shortcoming by performing at least one of the possible actions included in the human-readable feedback.