<jupyter_start><jupyter_text>![title](in.jpg) Innoplexus Online Hiring Hackathon: Saving lives with AI Problem StatementClinical studies often require detailed patients’ information documented in clinical narratives. **Named Entity Recognition (NER) is a fundamental Natural Language Processing (NLP) task to extract entities of interest (e.g., disease names, medication names and lab tests) from clinical narratives, thus to support clinical and translational research.** Clinical notes have been analyzed in greater detail to harness important information for clinical research and other healthcare operations, as they depict rich, detailed medical information.In this challenge, hackers are invited to extract all disease names from a given set of 20000 paragraphs/documents in the test set provided the labelled entities (diseases) for 30000 documents in the train set.For example, here is a sentence from a clinical report:*We compared the inter-day reproducibility of post-occlusive **reactive hyperemia** (PORH) assessed by single-point laser Doppler flowmetry (LDF) and laser speckle contrast analysis (LSCI).*In the sentence given, **reactive hyperemia (in bold)** is the named entity with the type disease/indication.  Data DescriptionThe train file has the following structure: |Variable | Definition||---|---||id|	Unique ID for a token/word||Doc_ID	|Unique ID for a Document/Paragraph||Sent_ID|	Unique ID for a Sentence||Word	|Exact word/token||tag	(Target)| Named Entity Tag  |The target 'tag' follows the **Inside-outside-beginning (IOB)** tagging format. The IOB format (short for inside, outside, beginning) is a common tagging format for tagging tokens in named-entity recognition.**The B-indications (beginning) tag indicates that the token is the beginning of a disease entity (disease name in this case)An I-indications (inside) tag indicates that the token is inside an entityAn O (outside) tag indicates that a token is outside a disease entity** **Example**For more clarity, let's look at the same sample in the given tabular format, each row here corresponds to a word/token:The disease **'reactive hyperemia'** is labelled using **'B-indications'** for the word **'reactive'** and **'I-indications'** for the word **'hypermia'**. All the other words that are outside **'reactive hyperemia'** are labelled with **'O'.** Evaluation MetricThe evaluation for this contest is based on modified F1-Score as explained below:Suppose the ground truth has the following entities (mentioned in square brackets) for the given sentence**[Malaria] and [Yellow Fever] remain more deadly than [Hepatitis B] today**This has 3 entities.Supposing the actual prediction has the following**[Malaria] [and] [Yellow] Fever remain more deadly than Hepatitis B [today]**We have an exact match for Malaria, false positives for and and today, a false negative for Hepatitis B and a substring match for Yellow. We compute precision and recall by first defining matching criteria. We are also trying to reward partial match here and not just exact entity match.Here, True positives are of 2 types - Exact match and partial match and we are giving a weight of 1 to Exact Match and 0.5 to partial match. The computations are as follows:Exact Match = 1 (Malaria) and Partial Match = 1 ( Yellow which overlaps Yellow Fever), False Positives =2 (and, and today), False Negatives = 1 (Hepatitis B)**Precision** = (Exact Match + 0.5 * Partial Match) / (Exact Match + Partial Match + False Positives) = (1 + 0.5)/(1+1+2) = 0.375**Recall** = (Exact Match + 0.5 * Partial Match) / (Exact Match + Partial Match + False Negatives) = (1 + 0.5)/(1+1+1) = 0.50**F1 Score** = (2 * Precision * Recall)/(Precision + Recall) = 0.428The counts of exact match, partial match, false positives and false negatives is summed across all sentences in the test set and overall F1 Score is the leaderboard score.Please find the script for the evaluation metric implemented in Python at this [link](https://gist.github.com/frenzy2106/3a12b7fefeb33941edea45d881d6f81a)<jupyter_code>import numpy as np
import pandas as pd
train=pd.read_csv('train.csv')
test=pd.read_csv('test.csv')
s=pd.read_csv('sample_submission.csv')
train.head()
test.head()
s.head()
train['tag'].value_counts()
train.dropna(inplace=True)
print(train.shape)
# df=train[train['Doc_ID']<=20000]
df=train.copy()
df.tag.value_counts()
test['Word'].fillna('NA',inplace=True)
from sklearn.metrics import classification_report
import sklearn_crfsuite
from sklearn_crfsuite import scorers
from sklearn_crfsuite import metrics
from collections import Counter<jupyter_output><empty_output><jupyter_text>Cleaning<jupyter_code>import re
def clean(v):
    v=str(v)
    r=''
    
    if len(v)!=1 and (v[-1]=='.' or v.find(':')!=-1 or v.find("'")!=-1 or v.find(",")!=-1):
        r=re.sub(r'[^\w\s]','',v)
    else:
        r=v
    if r=='':
        return ','
    return r
    
df['Word']=df['Word'].apply(clean)

test['Word']=test['Word'].apply(clean)
df.shape<jupyter_output><empty_output><jupyter_text>**Grouping by Sent_ID and checking if the tokens formed are in unison with training rows** **Extracting pos tagging from the tokens using nltk.pos_tag** Train<jupyter_code>from nltk import word_tokenize, pos_tag, ne_chunk
agg_func = lambda s: " ".join([w for w in s['Word'].values.tolist()])
sentids=df.groupby(['Sent_ID']).apply(agg_func)
post=[]
for lll in sentids:
    words = word_tokenize(lll)
    if len(lll.split(" "))!=len(words):
        print(lll)
        print(words)
        raise Exception('heheheh')
    post.extend([q[1] for q in pos_tag(words)])
    

len(post)
from nltk import word_tokenize, pos_tag, ne_chunk<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>test=pd.read_csv('test.csv')
test['Word'].fillna('NA',inplace=True)
test['Word']=test['Word'].apply(clean)

from nltk import word_tokenize, pos_tag, ne_chunk
pos=[]
tsentids=test.groupby(['Sent_ID']).apply(agg_func)
for ll in tsentids:
    
    pos.extend([q[1] for q in pos_tag(word_tokenize(ll))])
print(len(pos))
print(len(pos),test.shape)
import gc
gc.collect()
print(len(pos),pos)
df['pos']=post
test['pos']=pos
# post
df.head()<jupyter_output><empty_output><jupyter_text>** Generating conlltags with help of nltk.chunk and making another feature with  IOB tagging as 'O' / 'I' / 'B'**Train<jupyter_code>from nltk.chunk import conlltags2tree, tree2conlltags
from nltk import word_tokenize, pos_tag, ne_chunk


from nltk import word_tokenize, pos_tag, ne_chunk
pos_iob=[]
for lx in sentids:
    words = word_tokenize(lx)
    if len(lx.split(" "))!=len(words):
#         print(i)
        print(lll)
        print(words)
        raise Exception('heheheh')
    pos_iob.extend([q[2][0] for q in tree2conlltags(ne_chunk(pos_tag(words)))])
len(pos_iob)<jupyter_output><empty_output><jupyter_text>Test<jupyter_code>from nltk.chunk import conlltags2tree, tree2conlltags
from nltk import word_tokenize, pos_tag, ne_chunk


from nltk import word_tokenize, pos_tag, ne_chunk
# agg_func = lambda s: " ".join([w for w in s['Word'].values.tolist()])
# sentids=df.groupby(['Sent_ID']).apply(agg_func)
post_iob=[]
for lxx in tsentids:
    words = word_tokenize(lxx)
    if len(lxx.split(" "))!=len(words):
#         print(i)
        print(lxx)
        print(words)
        raise Exception('heheheh')
    post_iob.extend([q[2][0] for q in tree2conlltags(ne_chunk(pos_tag(words)))])<jupyter_output><empty_output><jupyter_text>Saving data<jupyter_code>df['tree_iob']=pos_iob
test['tree_iob']=post_iob
df['tree_iob'].to_csv('train_treeiob.csv',index=False)
test['tree_iob'].to_csv('test_treeiob.csv',index=False)
test['tree_iob'].value_counts()
# X
df['pos'].to_csv('train_pos.csv',index=False)
test['pos'].to_csv('test_pos.csv',index=False)
# np.save('sent_X.npy',X)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12015901513208516, 'fraction_numerical': 0.011926134906386252, 'mean_word_length': 5.482959268495428, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 29, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '534632', 'license': 'Apache-2.0', 'n_tokens_mistral': 2596, 'n_tokens_neox': 2459, 'n_words': 814, 'path': 'Generating_feat.ipynb', 'repo_name': 'rajat5ranjan/AV-Innoplexus-Hack-Saving-Lives-with-AI', 'size': '7798'}