<jupyter_start><jupyter_text>![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/Certification_Trainings/Healthcare/11.2.Pretrained_NER_Profiling_Pipelines.ipynb)<jupyter_code>import json, os
from google.colab import files

license_keys = files.upload()

with open(list(license_keys.keys())[0]) as f:
    license_keys = json.load(f)

# Defining license key-value pairs as local variables
locals().update(license_keys)

# Adding license key-value pairs to environment variables
os.environ.update(license_keys)
# Installing pyspark and spark-nlp
! pip install --upgrade -q pyspark==3.1.2 spark-nlp==$PUBLIC_VERSION

# Installing Spark NLP Healthcare
! pip install --upgrade -q spark-nlp-jsl==$JSL_VERSION  --extra-index-url https://pypi.johnsnowlabs.com/$SECRET
import json
import os
from pyspark.ml import Pipeline,PipelineModel
from pyspark.sql import SparkSession

from sparknlp.annotator import *
from sparknlp_jsl.annotator import *
from sparknlp.base import *
import sparknlp_jsl
import sparknlp

params = {"spark.driver.memory":"16G",
"spark.kryoserializer.buffer.max":"2000M",
"spark.driver.maxResultSize":"2000M"}

spark = sparknlp_jsl.start(license_keys['SECRET'],params=params)

print ("Spark NLP Version :", sparknlp.version())
print ("Spark NLP_JSL Version :", sparknlp_jsl.version())<jupyter_output>Spark NLP Version : 3.3.4
Spark NLP_JSL Version : 3.3.4<jupyter_text>Clinical NER Model Profiling Pretrained PipelineThis pipeline can be used to explore all the available pretrained NER models at once. When you run this pipeline over your text, you will end up with the predictions coming out of each pretrained clinical NER model trained with `embeddings_clinical`.|Clinical NER Model List||-||ner_ade_clinical||ner_posology_greedy||ner_risk_factors||jsl_ner_wip_clinical||ner_human_phenotype_gene_clinical||jsl_ner_wip_greedy_clinical||ner_cellular||ner_cancer_genetics||jsl_ner_wip_modifier_clinical||ner_drugs_greedy||ner_deid_sd_large||ner_diseases||nerdl_tumour_demo||ner_deid_subentity_augmented||ner_jsl_enriched||ner_genetic_variants||ner_bionlp||ner_measurements_clinical||ner_diseases_large||ner_radiology||ner_deid_augmented||ner_anatomy||ner_chemprot_clinical||ner_posology_experimental||ner_drugs||ner_deid_sd||ner_posology_large||ner_deid_large||ner_posology||ner_deidentify_dl||ner_deid_enriched||ner_bacterial_species||ner_drugs_large||ner_clinical_large||jsl_rd_ner_wip_greedy_clinical||ner_medmentions_coarse||ner_radiology_wip_clinical||ner_clinical||ner_chemicals||ner_deid_synthetic||ner_events_clinical||ner_posology_small||ner_anatomy_coarse||ner_human_phenotype_go_clinical||ner_jsl_slim||ner_jsl||ner_jsl_greedy||ner_events_admission_clinical|You can check [Models Hub](https://nlp.johnsnowlabs.com/models) page for more information about all these models and more.<jupyter_code>from sparknlp.pretrained import PretrainedPipeline

clinical_profiling_pipeline = PretrainedPipeline("ner_profiling_clinical", "en", "clinical/models")
text = '''A 28-year-old female with a history of gestational diabetes mellitus diagnosed eight years prior to presentation and subsequent type two diabetes mellitus ( T2DM ), one prior episode of HTG-induced pancreatitis three years prior to presentation , associated with an acute hepatitis , and obesity with a body mass index ( BMI ) of 33.5 kg/m2 , presented with a one-week history of polyuria , polydipsia , poor appetite , and vomiting .'''
clinical_result = clinical_profiling_pipeline.annotate(text)
clinical_result.keys()
print("Clinical Text: \n", text, "\n \nResults:\n")

for i in clinical_result.keys():
  print("{} : ".format(i), clinical_result[i])<jupyter_output>Clinical Text: 
 A 28-year-old female with a history of gestational diabetes mellitus diagnosed eight years prior to presentation and subsequent type two diabetes mellitus ( T2DM ), one prior episode of HTG-induced pancreatitis three years prior to presentation , associated with an acute hepatitis , and obesity with a body mass index ( BMI ) of 33.5 kg/m2 , presented with a one-week history of polyuria , polydipsia , poor appetite , and vomiting . 
 
Results:

ner_ade_clinical_chunks :  ['polydipsia', 'poor appetite', 'vomiting']
ner_deid_augmented :  ['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O']
ner_posology_greedy_chunks :  []
ner_radiology_wip_clinical :  ['O', 'O', 'O[...]<jupyter_text>**Lets show chunk results of NER models in a pandas dataframe.**<jupyter_code>import pandas as pd


def get_chunk_results(light_result):

  for i in light_result.keys():
    
    model_name = "_".join(i.split("_")[:-1])

    if (i == 'sentence') | (i == 'token'):
      
      continue
    
    if ("_chunks" not in i):
      
      continue

    elif len(light_result[i]) == 0:
      
      print("\n", "*"*20, model_name, "Model Results", "*"*20)
      print("No Result For This Model")


    else:

      sentences = []
      begins = []
      ends = []
      chunks = []
      entities = []
      confidences = []

      for n in (light_result[i]):

        sentences.append(n.metadata['sentence'])
        begins.append(n.begin)
        ends.append(n.end) 
        chunks.append(n.result)
        entities.append(n.metadata['entity'])
        confidences.append(n.metadata['confidence'])

        
      df = pd.DataFrame({'sentence':sentences, 'begin': begins, 'end': ends, 'chunks': chunks, 'entity': entities, 'confidence': confidences})

      print("\n \n", "*"*20, model_name, "Model Results", "*"*20)
      display(df)
text2 = """
At this time, chest tube placement for drainage of the fluid occurred and thoracoscopy with fluid biopsies, which were performed, which revealed epithelioid malignant mesothelioma. 
The patient was then stained with a PET CT, which showed extensive uptake in the chest, bilateral pleural pericardial effusions, and lymphadenopathy. 
She also had acidic fluid, pectoral and intramammary lymph nodes and uptake in L4 with SUV of 4. This was consistent with stage III disease. Her repeat echocardiogram showed an ejection fraction of 45% to 49%. 
She was transferred to Oncology service and started on chemotherapy on September 1, 2007 with cisplatin 75 mg/centimeter squared equaling 109 mg IV piggyback over 2 hours on September 1, 2007
"""
clinical_full_result = clinical_profiling_pipeline.fullAnnotate(text2)[0]
clinical_full_result['ner_jsl_chunks']
print("Clinical Text:\n", text2)
get_chunk_results(clinical_full_result)<jupyter_output>******************** ner_ade_clinical Model Results ********************<jupyter_text>**Now we will show all NER labels of tokens in the same dataframe.**<jupyter_code>import pandas as pd

def get_token_results(light_result):

  tokens = [j.result for j in light_result["token"]]
  sentences = [j.metadata["sentence"] for j in light_result["token"]]
  begins = [j.begin for j in light_result["token"]]
  ends = [j.end for j in light_result["token"]]
  model_list = [ a for a in light_result.keys() if (a not in ["sentence", "token"] and "_chunks" not in a)]

  df = pd.DataFrame({'sentence':sentences, 'begin': begins, 'end': ends, 'token':tokens})

  for model_name in model_list:
    
    temp_df = pd.DataFrame(light_result[model_name])
    temp_df["jsl_label"] = temp_df.iloc[:,0].apply(lambda x : x.result)
    temp_df = temp_df[["jsl_label"]]

    # temp_df = get_ner_result(model_name)
    temp_df.columns = [model_name]
    df = pd.concat([df, temp_df], axis=1)
    
  return df
get_token_results(clinical_full_result)<jupyter_output><empty_output><jupyter_text>BioBert NER Model Profiling Pretrained PipelineThis pipeline can be used to explore all the available pretrained NER models at once. When you run this pipeline over your text, you will end up with the predictions coming out of each pretrained clinical NER model trained with `biobert_pubmed_base_cased`.|BioBert NER Model List||-||ner_cellular_biobert||ner_diseases_biobert||ner_events_biobert||ner_bionlp_biobert||ner_jsl_greedy_biobert||ner_jsl_biobert||ner_anatomy_biobert||ner_jsl_enriched_biobert||ner_human_phenotype_go_biobert||ner_deid_biobert||ner_deid_enriched_biobert||ner_clinical_biobert||ner_anatomy_coarse_biobert||ner_human_phenotype_gene_biobert||ner_posology_large_biobert||jsl_rd_ner_wip_greedy_biobert||ner_posology_biobert||jsl_ner_wip_greedy_biobert||ner_chemprot_biobert||ner_ade_biobert||ner_risk_factors_biobert|You can check [Models Hub](https://nlp.johnsnowlabs.com/models) page for more information about all these models and more.<jupyter_code>from sparknlp.pretrained import PretrainedPipeline

biobert_profiling_pipeline = PretrainedPipeline("ner_profiling_biobert", "en", "clinical/models")
bio_result = biobert_profiling_pipeline.annotate(text)
bio_result.keys()
print("Clinical Text: \n", text, "\n \nResults:\n")

for i in bio_result.keys():
  print("{} : ".format(i), bio_result[i])
bio_full_result = biobert_profiling_pipeline.fullAnnotate(text2)[0]
bio_full_result['ner_jsl_biobert_chunks']
print("Clinical Text:\n", text2)<jupyter_output>Clinical Text:
 
At this time, chest tube placement for drainage of the fluid occurred and thoracoscopy with fluid biopsies, which were performed, which revealed epithelioid malignant mesothelioma. 
The patient was then stained with a PET CT, which showed extensive uptake in the chest, bilateral pleural pericardial effusions, and lymphadenopathy. 
She also had acidic fluid, pectoral and intramammary lymph nodes and uptake in L4 with SUV of 4. This was consistent with stage III disease. Her repeat echocardiogram showed an ejection fraction of 45% to 49%. 
She was transferred to Oncology service and started on chemotherapy on September 1, 2007 with cisplatin 75 mg/centimeter squared equaling 109 mg IV piggyback over 2 hours on September 1, 2007<jupyter_text>**Show NER chunks in a pandas dataframe.**<jupyter_code>get_chunk_results(bio_full_result)<jupyter_output>******************** ner_cellular_biobert Model Results ********************
No Result For This Model

 
 ******************** ner_diseases_biobert Model Results ********************<jupyter_text>**Show NER label results in a pandas dataframe.**<jupyter_code>get_token_results(clinical_full_result)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12574850299401197, 'fraction_numerical': 0.0093562874251497, 'mean_word_length': 6.04614370468029, 'pattern_counts': {'":': 3, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '54914', 'license': 'Apache-2.0', 'n_tokens_mistral': 3787, 'n_tokens_neox': 3429, 'n_words': 928, 'path': 'tutorials/Certification_Trainings/Healthcare/11.2.Pretrained_NER_Profiling_Pipelines.ipynb', 'repo_name': 'Pandinosaurus/spark-nlp-workshop', 'size': '10688'}