I am trying to write a script of Python code, for entity extraction and resolution.The excerpts of the algorithm:It is trying to extract the entity as PoS Tag with Hidden Markov Model(HMM).After training and testing, application data is given to tagger.It is extracting the user defined entities.A different Naive Bayes Classifier (NBC) is trained to identify the sentences with relations like, 'APPO'->for apposition, 'Alias'->for alias, 'DT'-> for Determiner, etc. The data of HMM and NBC are different.The application data of HMM is the application data of NBC, post tagging or identifying of entities.I have used mainly Relationship extraction procedure suggested by Daniel Zurafsky to resolve entities.Based on this idea, I wrote the following script:import nltkfrom nltk.corpus.reader import TaggedCorpusReaderfrom sklearn import cross_validationfrom nltk.tokenize import word_tokenizefrom itertools import chaindef ERR():    #HMM PORTION    reader = TaggedCorpusReader('.', r'.*\.pos')    files=reader.fileids()    tagcorpus=reader.tagged_sents('TotalData.pos')    testcorpus=reader.tagged_sents('TestHMM.pos')    app_corp=reader.sents('HMMApplication.pos')    len_tagcorp=len(tagcorpus)    print The Length of the Annotated Training Corpus,len_tagcorp    #train_corp=tagcorpus[:1380]    train_corp=tagcorpus    test_corp=testcorpus    hmm_tagger=nltk.HiddenMarkovModelTagger.train(train_corp)    hmm_score=hmm_tagger.evaluate(test_corp)    print HMM Score Is:,hmm_score    #NAIVE BAYES TRAINING    nbf1=open(/python27/classfdata/Classif1data/AppoNew3.txt,r).read().lower() #New Set1    nbf2=open(/python27/classfdata/Classif1data/AliasNew3.txt,r).read().lower() #New Set1    nbf3=open(/python27/classfdata/Classif1data/DTNew3.txt,r).read().lower() #New Set1    nbf4=open(/python27/classfdata/Classif1data/PRNNew3.txt,r).read().lower() #New Set1    nbf5=open(/python27/classfdata/Classif1data/SMNew3.txt,r).read().lower() #New Set1    nbf6=open(/python27/classfdata/Classif1data/AmbNew1.txt,r).read().lower() #AMBIGUITY    nbf7=open(/python27/classfdata/Classif1data/SMAMB1.txt,r).read().lower() #AMBIGUITY     list2=[(nbf1,'APPO'),(nbf2,'Alias'),(nbf3,'DT'),(nbf4,'PRNAMB'),(nbf5,'SM'),(nbf6,'AMB'),(nbf7,'SMAMB')]    len_list2=len(list2)    train_sents=list2[:5]    test_sents=list2[3:]    all_words = set(word.lower() for passage in list2 for word in word_tokenize(passage[0]))    t = [({word: (word in word_tokenize(x[0])) for word in all_words}, x[1]) for x in train_sents]    t1=[({word: (word in word_tokenize(x[0])) for word in all_words}, x[1]) for x in test_sents]    classifier = nltk.NaiveBayesClassifier.train(t)    print NLTK NAIVE BAYES ACCURACY:,(nltk.classify.accuracy(classifier, t1))    for i in app_corp:        classif=hmm_tagger.tag(i)        #FLATTENING THE LIST OF TUPLES        list3=list(chain(*classif))        try:            a3=[w.replace('NA', '/NA').replace('COMP','/COMP').replace('COMPSu','/COMPSu').replace('EXET','/EXET').replace('AMT','/AMT').replace('ArtDef','/ArtDef').replace('AdjDef','/AdjDef').replace('AdjLoc','/AdjLoc').replace('PERSB','/PERSB').replace('DPERS','/DPERS').replace('DT','/DT').replace('PERSPL','/PERSPL').replace('EXET','/EXET').replace('LOC','/LOC').replace('LOCC','/LOCC').replace('TRM','/TRM').replace('ORGRel','/ORGRel').replace('ORGP','/ORGP').replace('PERSRel','/PERSRel').replace('PERSREM','/PERSREM').replace('PRNM3PAS','/PRNM3PAS').replace('PRNneu3PAS','/PRNneu3PAS').replace('PRNM3PAS','/PRNM3PAS').replace('PRNneu3PAS','/PRNneu3PAS').replace('LOCDi','/LOCDi').replace('LOCPOS','/LOCPOS').replace('ORGREL','/ORGREL').replace('ARTDEF','/ARTDEF').replace('STR','/STR').replace('GM','/GM').replace('PERS','/PERS') for w in list3]            a4= .join(a3).replace(' /','/').replace('//','/')            #CLASSIFYING            x1=a4            print The Sentence Is:,x1            test_sent_features1 = {word.lower(): (word in word_tokenize(x1.lower())) for word in all_words}            result=classifier.classify(test_sent_features1)            print CLASSIFIER RESULT:,result        except AttributeError:            print ErrorAm I being able to tackle the problem? Could I combine HMM and NBC fine?NB: Please see  link for sample HMM training data. NBC training data is very close in nature though labels are quite different.
Entity resolution with NLTK
python;machine learning;natural language processing;markov chain