I am trying to train a question-answer system, where I am trying to group similar questions, and identify the most apt response. The program should identify the intent/focus.To do it, I have tagged the questions as NER so that I may know which intent it has, like PERSON or LOCATION etc. I am trying to group similar questions in a classifier, where NE tagged questions are X, and answers are Y.I tried to write the following code. I am trying to paste the .pos file data in Pastebin. The code is running on toy-sized data thus I could not keep proper training/testing  data partitions. But it seems giving me satisfactory results.Please suggest how I can write the code beautifully.import nltkimport collectionsimport nltk.metricsfrom nltk.corpus.reader import TaggedCorpusReaderfrom nltk.tokenize import word_tokenizefrom itertools import chainfrom nltk.classify import NaiveBayesClassifier as nbcfrom itertools import chain#HMM reader = TaggedCorpusReader('.', r'.*\.pos')f1=reader.fileids()print f1sents=reader.tagged_sents(fileids='Monishnew1.pos')ls=len(sents)print lstrain_sents=sents[:7]test_sents=sents[7:]class_sents=reader.sents(fileids='Monishnew1.pos')appli_sent1=class_sents[5]print Sentence Is:,appli_sent1 hmm_tagger=nltk.HiddenMarkovModelTagger.train(train_sents)test=hmm_tagger.test(test_sents)print testclassf1=hmm_tagger.tag(appli_sent1)print classf1joiner1 = /.joinclassf2=map(joiner1, classf1)print classf2classf3= .join(classf2)print classf3#NBC#CLASSIFYING THE ANSWERtraining_data=[('Hello/GREET',' How can I assist you'),               ('Hi/GREET','How can I assist you'),               ('want/NA to/NA speak/NA to/NA customer/GREET care/GREETC','How can I assist you'),               ('issue/NA with/NA the/NA report/REPORT', ' Please tell me your exact problem'),('want/NA to/NA see/NA my/NA test/REPORTP report/REPORT','Sure Please share the lab number'),('know/NA my/NA report/REPORT test/REPORTC done/NA today/NAgot/NA the/NA message/NA for/NA the/NA report/REPORT generation/REPORTC but/NA unable/NA to/NA see/NA','Apologies for the inconvenience caused Please share the lab number'),('the/NA report/REPORT 224622415/REPVW  231223456/REPVW how/REPVW to/NA check/REPVW it/NA','please share you lab number to check this'),('thanks/CLOSEok/CLOSEthank/CLOSE you/CLOSEC When/CLOSE will/NA I/NA get/NA the/NA call/CLOSE','is there anything else I can help you with'),('I/NA will/NA wait/NA for/NA the/NA call/CLOSE','is there anything else I can help you with'),('when/CLOSE call/CLOSE me/NA back/NA','you will get a call in the next 12 hours Is there anything else I can help you with')]print training_datavocabulary = set(chain(*[word_tokenize(i[0].lower()) for i in training_data]))feature_set = [({i:(i in word_tokenize(sentence.lower())) for i in vocabulary},tag) for sentence, tag in training_data]train_set, test_set = feature_set[:5], feature_set[5:]classifier = nbc.train(train_set)refsets = collections.defaultdict(set)testsets = collections.defaultdict(set)for i, (feats, label) in enumerate(test_set):    refsets[label].add(i)    observed = classifier.classify(feats)    testsets[observed].add(i)test_sentence=classf3featurized_test_sentence = {i:(i in word_tokenize(test_sentence.lower())) for i in vocabulary}cat1=classifier.classify(featurized_test_sentence)print The Answer Is:,cat1
Nave Bayes classifier to group questions by intent
python;machine learning;natural language processing