<jupyter_start><jupyter_text>![title](in.jpg) Innoplexus Online Hiring Hackathon: Saving lives with AI Problem StatementClinical studies often require detailed patients’ information documented in clinical narratives. **Named Entity Recognition (NER) is a fundamental Natural Language Processing (NLP) task to extract entities of interest (e.g., disease names, medication names and lab tests) from clinical narratives, thus to support clinical and translational research.** Clinical notes have been analyzed in greater detail to harness important information for clinical research and other healthcare operations, as they depict rich, detailed medical information.In this challenge, hackers are invited to extract all disease names from a given set of 20000 paragraphs/documents in the test set provided the labelled entities (diseases) for 30000 documents in the train set.For example, here is a sentence from a clinical report:*We compared the inter-day reproducibility of post-occlusive **reactive hyperemia** (PORH) assessed by single-point laser Doppler flowmetry (LDF) and laser speckle contrast analysis (LSCI).*In the sentence given, **reactive hyperemia (in bold)** is the named entity with the type disease/indication.  Data DescriptionThe train file has the following structure: |Variable | Definition||---|---||id|	Unique ID for a token/word||Doc_ID	|Unique ID for a Document/Paragraph||Sent_ID|	Unique ID for a Sentence||Word	|Exact word/token||tag	(Target)| Named Entity Tag  |The target 'tag' follows the **Inside-outside-beginning (IOB)** tagging format. The IOB format (short for inside, outside, beginning) is a common tagging format for tagging tokens in named-entity recognition.**The B-indications (beginning) tag indicates that the token is the beginning of a disease entity (disease name in this case)An I-indications (inside) tag indicates that the token is inside an entityAn O (outside) tag indicates that a token is outside a disease entity** **Example**For more clarity, let's look at the same sample in the given tabular format, each row here corresponds to a word/token:The disease **'reactive hyperemia'** is labelled using **'B-indications'** for the word **'reactive'** and **'I-indications'** for the word **'hypermia'**. All the other words that are outside **'reactive hyperemia'** are labelled with **'O'.** Evaluation MetricThe evaluation for this contest is based on modified F1-Score as explained below:Suppose the ground truth has the following entities (mentioned in square brackets) for the given sentence**[Malaria] and [Yellow Fever] remain more deadly than [Hepatitis B] today**This has 3 entities.Supposing the actual prediction has the following**[Malaria] [and] [Yellow] Fever remain more deadly than Hepatitis B [today]**We have an exact match for Malaria, false positives for and and today, a false negative for Hepatitis B and a substring match for Yellow. We compute precision and recall by first defining matching criteria. We are also trying to reward partial match here and not just exact entity match.Here, True positives are of 2 types - Exact match and partial match and we are giving a weight of 1 to Exact Match and 0.5 to partial match. The computations are as follows:Exact Match = 1 (Malaria) and Partial Match = 1 ( Yellow which overlaps Yellow Fever), False Positives =2 (and, and today), False Negatives = 1 (Hepatitis B)**Precision** = (Exact Match + 0.5 * Partial Match) / (Exact Match + Partial Match + False Positives) = (1 + 0.5)/(1+1+2) = 0.375**Recall** = (Exact Match + 0.5 * Partial Match) / (Exact Match + Partial Match + False Negatives) = (1 + 0.5)/(1+1+1) = 0.50**F1 Score** = (2 * Precision * Recall)/(Precision + Recall) = 0.428The counts of exact match, partial match, false positives and false negatives is summed across all sentences in the test set and overall F1 Score is the leaderboard score.Please find the script for the evaluation metric implemented in Python at this [link](https://gist.github.com/frenzy2106/3a12b7fefeb33941edea45d881d6f81a)<jupyter_code>import numpy as np
import pandas as pd
train=pd.read_csv('train.csv')
test=pd.read_csv('test.csv')
s=pd.read_csv('sample_submission.csv')



train.head()<jupyter_output><empty_output><jupyter_text>Fetching Data<jupyter_code>trainiob=pd.read_csv('train_treeiob.csv')

testiob=pd.read_csv('test_treeiob.csv')

trainpos=pd.read_csv('train_pos.csv')
testpos=pd.read_csv('test_pos.csv')



train['pos']=trainpos['pos']
train['tree_iob']=trainiob['tree_iob']


test['pos']=testpos['pos']
test['tree_iob']=testiob['tree_iob']

train.head()

df=train.copy()
dftest=test.copy()
del train
del test<jupyter_output><empty_output><jupyter_text>**Cleaning**<jupyter_code>import re
def clean(v):
    v=str(v)
    r=''
    
    if len(v)!=1 and (v[-1]=='.' or v.find(':')!=-1 or v.find("'")!=-1 or v.find(",")!=-1):
        r=re.sub(r'[^\w\s]','',v)
    else:
        r=v
    if r=='':
        return ','
    return r
    
df['Word']=df['Word'].apply(clean)
dftest['Word']=dftest['Word'].apply(clean)
df.head()
from nltk.corpus import stopwords
from string import punctuation
st=stopwords.words('english')
df.head()
class SentenceGetter(object):
    
    def __init__(self, data):
        self.n_sent = 1
        self.data = data
        self.empty = False
        agg_func = lambda s: [(w,p,e, t) for w,p,e, t in zip(s['Word'].astype(str).values.tolist(), 
                                                       s['pos'].astype(str).values.tolist(), s['tree_iob'].values.tolist(),
                                                           s['tag'].values.tolist())]
        self.grouped = self.data.groupby(['Doc_ID','Sent_ID']).apply(agg_func)
        self.sentences = [s for s in self.grouped]
        
    def get_next(self):
        try: 
            s = self.grouped['Sentence: {}'.format(self.n_sent)]
            self.n_sent += 1
            return s 
        except:
            return None
getter = SentenceGetter(df)
sentences = getter.sentences
def word2features(sent, i):
    word = sent[i][0]
    postag = sent[i][1]
    tr_iob = sent[i][2]
    
    features = {
        'bias': 1.0,
        'word.lower()': word.lower(),
        'word[-3:]': word[-3:],
        'word[-2:]': word[-2:],
        'word.isupper()': word.isupper(),
        'word.istitle()': word.istitle(),
        'word.isdigit()': word.isdigit(),
        'word.isidentifier()':word.isidentifier(),
        'postag': postag,
        'postag[:2]': postag[:2], 
        'tree_iob':tr_iob,
    }
    if i > 0:
        word1 = sent[i-1][0]
        postag1 = sent[i-1][1]
        tr_iob1 = sent[i-1][2]
        features.update({
            '-1:word.lower()': word1.lower(),
            '-1:word.istitle()': word1.istitle(),
            '-1:word.isupper()': word1.isupper(),
            '-1:postag': postag1,
            '-1:postag[:2]': postag1[:2],
            '-1:tree_iob':tr_iob1,
        })
    else:
        features['BOS'] = True
        
    if i < len(sent)-1:
        word1 = sent[i+1][0]
        postag1 = sent[i+1][1]
        tr_iob1=sent[i+1][2]
        features.update({
            '+1:word.lower()': word1.lower(),
            '+1:word.istitle()': word1.istitle(),
            '+1:word.isupper()': word1.isupper(),
            '+1:postag': postag1,
            '+1:postag[:2]': postag1[:2],
            '+1:tree_iob':tr_iob1,
        })
    else:
        features['EOS'] = True
                
    return features


def sent2features(sent):
    return [word2features(sent, i) for i in range(len(sent))]

def sent2labels(sent):
    return [label for token, postag,tiob, label in sent]

def sent2tokens(sent):
    return [token for token, postag,tiob, label in sent]
from tqdm import tqdm_notebook as tqdm
X = [sent2features(s) for s in tqdm(sentences)]
y = [sent2labels(s) for s in tqdm(sentences)]
# from sklearn.feature_extraction import DictVectorizer
# from sklearn.feature_extraction.text import HashingVectorizer
# from sklearn.linear_model import Perceptron
from sklearn.model_selection import train_test_split
# from sklearn.linear_model import SGDClassifier
# from sklearn.linear_model import PassiveAggressiveClassifier
# from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import classification_report
X_train, X_test, y_train, y_test = train_test_split(X[:600], y[:600], test_size=0.25, random_state=0)
import sklearn_crfsuite
from sklearn_crfsuite import scorers
from sklearn_crfsuite import metrics
from collections import Counter
import scipy.stats
from sklearn.metrics import make_scorer
from sklearn.cross_validation import cross_val_score
from sklearn.grid_search import RandomizedSearchCV<jupyter_output>C:\Users\Acer\Anaconda3\lib\site-packages\sklearn\cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)
C:\Users\Acer\Anaconda3\lib\site-packages\sklearn\grid_search.py:42: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. This module will be removed in 0.20.
  DeprecationWarning)<jupyter_text>Testing and Evaluating ModelFor only 600 data<jupyter_code>crf = sklearn_crfsuite.CRF(
    algorithm='lbfgs',
    c1=0.001,
    c2=0.001,
    max_iterations=100,
    all_possible_transitions=True,verbose=True
)
crf.fit(X_train, y_train)
# labels = list(crf.classes_)
# labels.remove('O')
labels=['B-indications','I-indications']
crf.classes_
from sklearn_crfsuite import metrics
y_pred = crf.predict(X_test)
print(metrics.flat_classification_report(y_test, y_pred, labels = labels))
metrics.flat_f1_score(y_test, y_pred,average='weighted', labels=labels)

crf = sklearn_crfsuite.CRF(
    algorithm='lbfgs',
    c1=0.01,
    c2=0.01,
    max_iterations=100,
    all_possible_transitions=True,verbose=True
)
crf.fit(X, y)
import pickle
pickle.dump(crf,open('crf_model.sav','wb'))
class SentenceGetterTest(object):
    
    def __init__(self, data):
        self.n_sent = 1
        self.data = data
        self.empty = False
        agg_func = lambda s: [(w,p,e) for w,p,e in zip(s['Word'].astype(str).values.tolist(), 
                                                       s['pos'].astype(str).values.tolist(), s['tree_iob'].values.tolist())]
        self.grouped = self.data.groupby(['Doc_ID','Sent_ID']).apply(agg_func)
        self.sentences = [s for s in self.grouped]
        
    def get_next(self):
        try: 
            s = self.grouped['Sentence: {}'.format(self.n_sent)]
            self.n_sent += 1
            return s 
        except:
            return None
getter_test = SentenceGetterTest(dftest)
test_sentences = getter_test.sentences
test_X = [sent2features(s) for s in tqdm(test_sentences)]
y_predtest = crf.predict(test_X)
z=[]
for zz in y_predtest:
    z.extend(zz)
print(len(z))
s=pd.read_csv('sample_submission.csv')
s['tag']=z
s['tag'].value_counts()
s.to_csv('s11_mainCRFft2.csv',index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12063294515068006, 'fraction_numerical': 0.018579429282602897, 'mean_word_length': 3.7468354430379747, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '556798', 'license': 'Apache-2.0', 'n_tokens_mistral': 3722, 'n_tokens_neox': 3520, 'n_words': 1053, 'path': 'Final_Submission.ipynb', 'repo_name': 'rajat5ranjan/AV-Innoplexus-Hack-Saving-Lives-with-AI', 'size': '11249'}