<jupyter_start><jupyter_text>Named Entity Recognition for Healthcare with SparkNLP NerDL and NerCRF - Data Preparation and Model Evaluation  Why is Data Preparation Important? It's simple to use a pretrained named entity recognition model, but sometimes you need to train your own model to get the best results. This tutorial will show you how to prepare your healthcare training data and train your own NER model using Python and SparkNLP. SparkNLP NerDL has cutting edge scores with the BC2GM dataset (Micro-average F1: 0.87) and other benchmark datasets. You need to use licensed SparkNLP Clinical embeddings to get those cutting edge scores on healthcare data, but Glove embeddings still do great. I'll show you how to train and evaluate your NerCRF and NerDL models on the BC5CDR-Chem dataset using Glove embeddings.  Preparing the Training Data To train a NerCRF or NerDL model, you will need to put your tokens and entity labels into a space-separated format called CoNLL. A CoNLL file puts each token of a sentence on a different line, and separates each sentence with an empty line. In the following Python example I will annotate one sentence and save it in CoNLL format.<jupyter_code>#Create some tokens
tokens=['An', 'apple', 'a', 'day', 'keeps', 'the', 'doctor', 'away', '.']

#Create part of speech labels or use a place-holder value like "NN".
pos_labels=['DT', 'NN', 'DT', 'NN', 'VBZ', 'DT', 'NN', 'RB', '.']

#Create some named entity labels. 'O' labels mean no named entity was found.
entity_labels=['B-Treatment','I-Treatment','I-Treatment','I-Treatment','O','O','O','O','O']<jupyter_output><empty_output><jupyter_text>Please notice the entity labels above. When an entity has more than one word, the label for the first word should begin with "B-" and the label for the following words should begin with "I-". Now let's save the tokens, parts-of-speech, and entity labels in CoNLL format.<jupyter_code>conll_lines=''

for token,pos,label in zip(tokens,pos_labels,entity_labels):
    
    conll_lines+="{} {} {} {}\n".format(token, pos, pos, label)

#Add another line break at the end of the sentence in order to create an empty line.
conll_lines+='\n'

#For this example I will print the lines instead of writing a .txt file.
print(conll_lines)<jupyter_output>An DT DT B-Treatment
apple NN NN I-Treatment
a DT DT I-Treatment
day NN NN I-Treatment
keeps VBZ VBZ O
the DT DT O
doctor NN NN O
away RB RB O
. . . O<jupyter_text>Please see the printed CoNLL above. "An" is the first word in "An apple a day" so it is labelled "B-Treatment", while "apple","a", and "day" are all labelled "I-Treatment". The words that are not "Treatments" are labelled with a capital "O". Here's another example of a sentence annotated in CoNLL format. The entity is "blood pressure".<jupyter_code>#Create some tokens
tokens=['I','checked','my','blood','pressure','this','morning','.']

#Create part-of-speech labels or use a place-holder value like 'NN'.
pos_labels=['PRP', 'VBD', 'PRP', 'NN', 'NN', 'DT', 'NN', '.']

#Create some named entity labels. 'O' labels mean no named entity was found
entity_labels=['O','O','O','B-Test','I-Test','O','O','O']
conll_lines=''

for token,pos,label in zip(tokens,pos_labels,entity_labels):
    
    conll_lines+="{} {} {} {}\n".format(token, pos, pos, label)

#Add another line break at the end of the sentence in order to create an empty line.
conll_lines+='\n'

#For this example I will print the lines instead of writing a .txt file.
print(conll_lines)<jupyter_output>I PRP PRP O
checked VBD VBD O
my PRP PRP O
blood NN NN B-Test
pressure NN NN I-Test
this DT DT O
morning NN NN O
. . . O<jupyter_text>As you can see above, 'blood' is the first word in the entity, so it is labelled "B-Test", while "pressure" is the second word in the entity so it is labelled "I-Test". We do this so the model can tell that "blood pressure" is one whole entity, rather than the two separate entities "blood" and "pressure. Now let's work with some real datasets. First we have to load the data.<jupyter_code>import os
! wget -O ncbi.tsv https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/resources/en/ner/conll-2003/NCBIdisease.tsv
! wget -O BC5CDRtrain.txt https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/resources/en/ner/conll-2003/CRFtrain_dev.txt
! wget -O BC5CDRtest.txt https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/resources/en/ner/conll-2003/CRFtest.txt<jupyter_output>--2020-07-20 14:02:17--  https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/resources/en/ner/conll-2003/NCBIdisease.tsv
Resolving s3.amazonaws.com (s3.amazonaws.com)... 172.16.17.32
Connecting to s3.amazonaws.com (s3.amazonaws.com)|172.16.17.32|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1060328 (1.0M) [text/tab-separated-values]
Saving to: ‘ncbi.tsv’

ncbi.tsv            100%[===================>]   1.01M  1.23MB/s    in 0.8s    

2020-07-20 14:02:18 (1.23 MB/s) - ‘ncbi.tsv’ saved [1060328/1060328]

--2020-07-20 14:02:18--  https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/resources/en/ner/conll-2003/CRFtrain_dev.txt
Resolving s3.amazonaws.com (s3.amazonaws.com)... 172.16.17.32
Connecting to s3.amazonaws.com (s3.amazonaws.com)|172.16.17.32|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3347168 (3.2M) [text/plain]
Saving to: ‘BC5CDRtrain.txt’

BC5CDRtrain.txt     100%[===================>]   3.19M  9.50MB/s    in 0.3[...]<jupyter_text>How to Convert a Pandas Dataframe to CoNLL Format In the next example I'll read from a Pandas dataframe and write a CoNLL file for NerDL. I'll use the sentence ID (sent_id) column to determine if I need to leave an empty line before a new sentence. Here are the first 5 lines of the dataframe:<jupyter_code>import pandas as pd
ncbi=pd.read_csv('ncbi.tsv',sep='\t')
ncbi.head()<jupyter_output><empty_output><jupyter_text>For NerDL the part-of-speech column is not used, but a CoNLL must still have a part of speech column. Add a part-of-speech column with 'NN' or some other placeholder as the only value. If you already have a part of speech column, you don't need to take this step.<jupyter_code>ncbi['pos']='NN'<jupyter_output><empty_output><jupyter_text>My Pandas dataframe is called 'ncbi' and I've added a part-of-speech column which I've called 'pos'. Now write a CoNLL file using the columns of the Pandas dataframe as input.<jupyter_code>conll_lines="-DOCSTART- -X- -X- -O-\n\n"
save=0

for sent, token, pos, label in zip(ncbi['sent_id'],ncbi['token'],ncbi['pos'],ncbi['entity_label']): 
    
# If the sentence ID has changed, that means we are starting a new sentence. We have to add an empty line.
    
    if save!=sent:
        conll_lines+='\n'
    
# Save the conll line
    
    conll_lines += "{} {} {} {}\n".format(token, pos, pos, label)
    
    save=sent
    

# Now print all of the lines to a text file

with open(file_loc,'w') as txtfile:
        
    for line in conll_lines:
        txtfile.write(line)

txtfile.close()<jupyter_output><empty_output><jupyter_text>If you look at the first 25 lines of the final CoNLL file, you'll see that rows containing only line breaks signal the beginning of a new sentence.<jupyter_code>with open(file_loc,'r') as f:
    lines=f.readlines()[0:25]
f.close()
lines<jupyter_output><empty_output><jupyter_text>Now let's see SparkNLPs cutting edge results! We'll train NerCRF and NerDL models on the BC5CDR-Chem benchmark dataset.  Training and Evaluating NerCRF NerCRF is a named entity recognition model in the SparkNLP library which is based on Conditional Random Fields. It requires part-of-speech for model training. To train a model with NerCRF, first import SparkNLP and start your Spark session. Then load the CoNLL.<jupyter_code>import sparknlp
from pyspark.ml import Pipeline
from pyspark.sql import SparkSession
from sparknlp.annotator import *
from sparknlp_jsl.annotator import *
from sparknlp.base import *
import sparknlp_jsl
import pyspark.sql.functions as F

spark = sparknlp.start()

spark
from sparknlp.training import CoNLL

file_loc='BC5CDRtrain.txt'
train = CoNLL().readDataset(spark, file_loc)
from pyspark.sql import functions as F

train.select(F.explode(F.arrays_zip('token.result','label.result')).alias("cols")) \
.select(F.expr("cols['0']").alias("token"),
        F.expr("cols['1']").alias("ground_truth")).groupBy('ground_truth').count()\
        .orderBy('count', ascending=False).show(100,truncate=False)<jupyter_output>+------------+------+
|ground_truth|count |
+------------+------+
|O           |221425|
|B-CHEM      |10550 |
|I-CHEM      |3648  |
+------------+------+<jupyter_text>I will add Glove embeddings to the dataset before Ner training, but if you want better results with your healthcare projects, use SparkNLP Clinical embeddings. First, set up your pipeline and fit your model to your training dataset. The fitting process could take some time.<jupyter_code>word_embeddings = WordEmbeddingsModel.pretrained('glove_100d')\
          .setInputCols(["document", "token"])\
          .setOutputCol("embeddings")

nerTagger = NerCrfApproach()\
    .setInputCols(["sentence", "token", "pos","embeddings"])\
    .setLabelColumn("label")\
    .setOutputCol("ner")\
    .setMaxEpochs(9)\
    
ner_pipeline = Pipeline(stages=[
          word_embeddings,
          nerTagger
 ])
ner_model = ner_pipeline.fit(train)<jupyter_output><empty_output><jupyter_text>Next add word embeddings to your test dataset and make your predictions.<jupyter_code>from sparknlp.training import CoNLL

file_loc='BC5CDRtest.txt'
test = CoNLL().readDataset(spark, file_loc)

test_data = word_embeddings.transform(test)

predictions = ner_model.transform(test_data)<jupyter_output><empty_output><jupyter_text>You can see all of your input and output columns in the final "predictions" dataframe, but I'll focus on the 'ner' column which contains the prediction, and the 'label' column which contains the ground truth. You can use sklearn.metrics classification_report to check the accuracy of the predictions using these 2 columns.<jupyter_code>from sklearn.metrics import classification_report
import pyspark.sql.functions as F

preds = predictions.select(F.explode(F.arrays_zip('token.result','label.result','ner.result')).alias("cols")) \
        .select(F.col('cols.0').alias("token"),
        F.col('cols.1').alias("label"),
        F.col('cols.2').alias("ner"))

preds.filter("ner!='O'").show(9)
#Convert the Spark dataframe to a Pandas dataframe.
import pandas as pd
preds_df=preds.toPandas()
print (classification_report(preds_df['label'], preds_df['ner']))<jupyter_output>precision    recall  f1-score   support

      B-CHEM       0.91      0.85      0.87      5385
      I-CHEM       0.72      0.84      0.77      1628
           O       0.99      0.99      0.99    117737

    accuracy                           0.98    124750
   macro avg       0.87      0.89      0.88    124750
weighted avg       0.98      0.98      0.98    124750<jupyter_text>Training and Evaluating NerDL NerDL is a deep learning named entity recognition model in the SparkNLP library which does not require training data to contain parts-of-speech. It is a Bidirectional LSTM-CNN. For a more detailed overview of training a model using NerDL, you can check out this [post](https://medium.com/r/?url=https%3A%2F%2Ftowardsdatascience.com%2Fnamed-entity-recognition-ner-with-bert-in-spark-nlp-874df20d1d77). We've already loaded the BC5CDR-Chem test and train datasets. Now I can show you how to add Glove embeddings and save the test data as a parquet file before NerDL model training<jupyter_code>word_embeddings = WordEmbeddingsModel.pretrained('glove_100d')\
          .setInputCols(["document", "token"])\
          .setOutputCol("embeddings")

test_data = word_embeddings.transform(test)

test_data.write.parquet('../test.parquet')<jupyter_output>glove_100d download started this may take some time.
Approximate size to download 145.3 MB
[OK!]<jupyter_text>Next set up the rest of the pipeline by adding the location of the test data parquet file and the folder where your Tensorflow graphs are located. Using ".setEvaluationLogExtended(True)" will output a more detailed model evaluation log. When you run the training, If you get an error for incompatible TF graph, use NerDL_Graph.ipynb located [here](https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/Certification_Trainings/Public/4.1_NerDL_Graph.ipynb) to create a graph using the parameters given in the error message. If you're having trouble with this part of NerDL model training, you should read this [post](https://medium.com/r/?url=https%3A%2F%2Ftowardsdatascience.com%2Fnamed-entity-recognition-ner-with-bert-in-spark-nlp-874df20d1d77).<jupyter_code>nerTagger = NerDLApproach()\
  .setInputCols(["sentence", "token", "embeddings"])\
  .setLabelColumn("label")\
  .setOutputCol("ner")\
  .setMaxEpochs(15)\
  .setLr(0.001)\
  .setPo(0.005)\
  .setBatchSize(32)\
  .setRandomSeed(0)\
  .setVerbose(1)\
  .setValidationSplit(0.2)\
  .setEvaluationLogExtended(True) \
  .setEnableOutputLogs(True)\
  .setIncludeConfidence(True)\
  .setGraphFolder('../tfgraphs')\
  .setTestDataset('../test.parquet')
                  
ner_pipeline = Pipeline(stages=[
          word_embeddings,
          nerTagger
 ])<jupyter_output><empty_output><jupyter_text>Even though the word_embeddings pipe is in a previous cell, it is still part of the pipeline. In the next cell I'll fit the model to the training set. This could take some time.<jupyter_code>%%time

ner_model = ner_pipeline.fit(train)<jupyter_output>CPU times: user 86.8 ms, sys: 34.4 ms, total: 121 ms
Wall time: 12min 2s<jupyter_text>You can find the final log at the top of the list here:<jupyter_code>! cd ~/annotator_logs && ls -lt<jupyter_output>total 280
-rw-rw-r-- 1 myilmaz myilmaz 13529 Jun 13 18:07 NerDLApproach_004e6626b3bb.log
-rw-rw-r-- 1 myilmaz myilmaz 17997 Jun 12 17:51 NerDLApproach_15b6d84b808b.log
-rw-rw-r-- 1 <NAME>  9819 Jun 11 20:00 NerDLApproach_31530d63198f.log
-rw-rw-r-- 1 <NAME>  1092 Jun 11 19:52 NerDLApproach_e5b8f13159eb.log
-rw-rw-r-- 1 <NAME>  1100 Jun 11 19:45 NerDLApproach_9e3eb5e1c0e9.log
-rw-rw-r-- 1 <NAME>  2007 Jun 11 10:11 NerDLApproach_7e555f3bb935.log
-rw-rw-r-- 1 <NAME>  2009 Jun 11 10:00 NerDLApproach_b61689d542fe.log
-rw-rw-r-- 1 <NAME> 12650 Jun 10 17:38 NerDLApproach_3a19217cbe4b.log
-rw-rw-r-- 1 <NAME>  1049 Jun  7 21:16 NerDLApproach_0560e20a620b.log
-rw-rw-r-- 1 <NAME> 55543 May 16 09:57 NerDLApproach_70a344517ff7.log
-rw-rw-r-- 1 <NAME> 55501 May 16 08:47 NerDLApproach_769f9d4b74d3.log
-rw-rw-r-- 1 myilmaz myilmaz 27771 May 16 08:15 NerDLApproach_acc76debb989.log
-rw-rw-r-- 1 myilmaz myilmaz [...]<jupyter_text>For each training epoch your extended log will print 2 sets of metrics, one for the validation dataset and one for the test dataset. (The metrics for the validation data is on the top). For each dataset there's a table showing true positives (tp), false positives (fp), false negatives (fn), precision, recall and f1 scores for each entity (except 'O'). Beneath this table you'll find the macro-average and micro-average precision, recall and f1 scores for the dataset. So if you're looking for the micro-average f1 score for the test data, you'll find it on the last line of the log for each epoch.<jupyter_code>!cat ~/annotator_logs/NerDLApproach_15b6d84b808b.log<jupyter_output>Name of the selected graph: /home/myilmaz/devel/mag/datasets/make_CoNLL/tutorial/tfgraphs/blstm_3_100_128_82.pb
Training started, trainExamples: 7313, labels: 3 chars: 81, 


Epoch: 0 started, learning rate: 0.001, dataset size: 7313
Done, 33.119487882 loss: 722.0155, batches: 231
Quality on validation dataset (20.0%), valExamples = 1828
time to finish evaluation: 3.284676026
label	 tp	 fp	 fn	 prec	 rec	 f1
B-CHEM	 1572	 294	 508	 0.8424437	 0.75576925	 0.7967562
I-CHEM	 349	 85	 501	 0.8041475	 0.41058823	 0.5436137
tp: 1921 fp: 379 fn: 1009 labels: 2
Macro-average	 prec: 0.8232956, rec: 0.58317876, f1: 0.6827405
Micro-average	 prec: 0.8352174, rec: 0.6556314, f1: 0.7346081
Quality on test dataset: 
time to finish evaluation: 8.096200166
label	 tp	 fp	 fn	 prec	 rec	 f1
B-CHEM	 3946	 743	 1439	 0.84154403	 0.7327762	 0.7834028
I-CHEM	 586	 232	 1042	 0.71638143	 0.35995087	 0.47914964
tp: 4532 fp: 975 fn: 2481 labels: 2
Macro-average	 prec: 0.77896273, rec: 0.5463[...]
{'fraction_non_alphanumeric': 0.11283738499969537, 'fraction_numerical': 0.056418692499847684, 'mean_word_length': 4.132582864290181, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 71, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '360782', 'license': 'Apache-2.0', 'n_tokens_mistral': 6303, 'n_tokens_neox': 5533, 'n_words': 1818, 'path': 'tutorials/blogposts/data_prep.ipynb', 'repo_name': 'fcivardi/spark-nlp-workshop', 'size': '16494'}