<jupyter_start><jupyter_text>Get Started with Spark NLP for Healthcare  Getting the keys and installation1. In order to get trial keys for Spark NLP for Healthcare, fill the form at https://www.johnsnowlabs.com/spark-nlp-try-free/ and you will get your keys to your email in a few minutes.2. On a new cluster or existing one  - add the following to the `Advanced Options -> Spark` tab, in `Spark.Config` box:    ```bash    spark.local.dir /var    spark.kryoserializer.buffer.max 1000M    spark.serializer org.apache.spark.serializer.KryoSerializer    ```  - add the following to the `Advanced Options -> Spark` tab, in `Environment Variables` box:    ```bash    AWS_ACCESS_KEY_ID=xxx    AWS_SECRET_ACCESS_KEY=yyy    SPARK_NLP_LICENSE=zzz    ```3. Download the followings with AWS CLI to your local computer    `$ aws s3 cp --region us-east-2 s3://pypi.johnsnowlabs.com/$jsl_secret/spark-nlp-jsl-$jsl_version.jar spark-nlp-jsl-$jsl_version.jar`    `$ aws s3 cp --region us-east-2 s3://pypi.johnsnowlabs.com/$jsl_secret/spark-nlp-jsl/spark_nlp_jsl-$jsl_version-py3-none-any.whl spark_nlp_jsl-$jsl_version-py3-none-any.whl` 4. In `Libraries` tab inside your cluster: - Install New -> PyPI -> `spark-nlp==$public_version` -> Install - Install New -> Maven -> Coordinates -> `com.johnsnowlabs.nlp:spark-nlp_2.12:$public_version` -> Install - add following jars for the Healthcare library that you downloaded above:        - Install New -> Python Whl -> upload `spark_nlp_jsl-$jsl_version-py3-none-any.whl`        - Install New -> Jar -> upload `spark-nlp-jsl-$jsl_version.jar`5. Now you can attach your notebook to the cluster and use Spark NLP!For more information, see   https://nlp.johnsnowlabs.com/docs/en/installdatabricks-support  https://nlp.johnsnowlabs.com/docs/en/licensed_installinstall-spark-nlp-for-healthcare-on-databricks  The follwing notebook is prepared and tested on **r2.2xlarge at 8.0 (includes Apache Spark 3.1.1, Scala 2.12)** on DatabricksIn order to get more detailed examples, please check this repository : https://github.com/JohnSnowLabs/spark-nlp-workshop/tree/master/tutorials/Certification_Trainings/Healthcare/databricks_notebooks Let's import the libraries which we will use in the following cells.<jupyter_code>import os
import json
import string
import numpy as np
import pandas as pd

import sparknlp
import sparknlp_jsl
from sparknlp.base import *
from sparknlp.util import *
from sparknlp.annotator import *
from sparknlp_jsl.base import *
from sparknlp_jsl.annotator import *
from sparknlp.pretrained import ResourceDownloader

from pyspark.sql import functions as F
from pyspark.ml import Pipeline, PipelineModel
from sparknlp.training import CoNLL

pd.set_option('max_colwidth', 100)
pd.set_option('display.max_columns', 100)  
pd.set_option('display.expand_frame_repr', False)

print('sparknlp.version : ',sparknlp.version())
print('sparknlp_jsl.version : ',sparknlp_jsl.version())

spark<jupyter_output><empty_output><jupyter_text>**Read Dataset**We will download a sample file and create a spark dataframe.<jupyter_code>! wget -q https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/resources/en/pubmed/pubmed_sample_text_small.csv
pubMedDF = spark.read.option("header", "true").csv("dbfs:/pubmed_sample_text_small.csv")
                
pubMedDF.show(truncate=100)<jupyter_output><empty_output><jupyter_text>1. Clinical NER PipelineWe will extract clinical entities from text by using `ner_clinical_large` model.<jupyter_code># Annotator that transforms a text column from dataframe into an Annotation ready for NLP
documentAssembler = DocumentAssembler()\
        .setInputCol("text")\
        .setOutputCol("document")

# Sentence Detector annotator, processes various sentences per line

#sentenceDetector = SentenceDetector()\
        #.setInputCols(["document"])\
        #.setOutputCol("sentence")
sentenceDetector = SentenceDetectorDLModel.pretrained("sentence_detector_dl_healthcare","en","clinical/models")\
        .setInputCols(["document"])\
        .setOutputCol("sentence")
 
# Tokenizer splits words in a relevant format for NLP
tokenizer = Tokenizer()\
        .setInputCols(["sentence"])\
        .setOutputCol("token")

# Clinical word embeddings trained on PubMED dataset
word_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical","en","clinical/models")\
        .setInputCols(["sentence","token"])\
        .setOutputCol("embeddings")

pos_tagger = PerceptronModel.pretrained("pos_clinical", "en", "clinical/models") \
        .setInputCols(["sentence", "token"])\
        .setOutputCol("pos_tags")

# NER model trained on i2b2 (sampled from MIMIC) dataset
clinical_ner = MedicalNerModel.pretrained("ner_clinical_large","en","clinical/models")\
        .setInputCols(["sentence","token","embeddings"])\
        .setOutputCol("ner")

ner_converter = NerConverter()\
        .setInputCols(["sentence","token","ner"])\
        .setOutputCol("ner_chunk")

nerPipeline = Pipeline(stages=[
        documentAssembler,
        sentenceDetector,
        tokenizer,
        word_embeddings,
        pos_tagger,
        clinical_ner,
        ner_converter])


empty_data = spark.createDataFrame([[""]]).toDF("text")

ner_model = nerPipeline.fit(empty_data)
ner_model.stages
clinical_ner.getClasses()
result = ner_model.transform(pubMedDF.limit(100))
result.show()
result.select("sentence.result").show(truncate=100)
from pyspark.sql import functions as F

result_df = result.select(F.explode(F.arrays_zip("token.result","ner.result")).alias("cols"))\
                  .select(F.expr("cols['0']").alias("token"),
                          F.expr("cols['1']").alias("ner_label"))

result_df.show(50, truncate=100)<jupyter_output><empty_output><jupyter_text>Lets count the ner_labels.<jupyter_code>result_df.select("token", "ner_label").groupBy('ner_label').count().orderBy('count', ascending=False).show(truncate=False)
result.select(F.explode(F.arrays_zip('ner_chunk.result', 'ner_chunk.begin', 'ner_chunk.end', 'ner_chunk.metadata')).alias("cols")) \
      .select(F.expr("cols['3']['sentence']").alias("sentence_id"),
              F.expr("cols['0']").alias("chunk"),
              F.expr("cols['1']").alias("begin"),
              F.expr("cols['2']").alias("end"),
              F.expr("cols['3']['entity']").alias("ner_label")).show(truncate=False)<jupyter_output><empty_output><jupyter_text>**We can also filter NER results to get specific entities by using `setWhiteList()` parameter. In this example we will get only `PROBLEM` entities.**<jupyter_code>ner_converter_filter = NerConverter()\
        .setInputCols(["sentence","token","ner"])\
        .setOutputCol("ner_chunk")\
        .setWhiteList(["PROBLEM"])

nerFilteredPipeline = Pipeline(stages=[
        documentAssembler,
        sentenceDetector,
        tokenizer,
        word_embeddings,
        clinical_ner,
        ner_converter_filter])


empty_data = spark.createDataFrame([[""]]).toDF("text")

ner_filtered_model = nerFilteredPipeline.fit(empty_data)
filtered_result = ner_filtered_model.transform(pubMedDF.limit(100))
filtered_result.select(F.explode(F.arrays_zip('ner_chunk.result', 'ner_chunk.begin', 'ner_chunk.end', 'ner_chunk.metadata')).alias("cols")) \
               .select(F.expr("cols['3']['sentence']").alias("sentence_id"),
                       F.expr("cols['0']").alias("chunk"),
                       F.expr("cols['1']").alias("begin"),
                       F.expr("cols['2']").alias("end"),
                       F.expr("cols['3']['entity']").alias("ner_label")).show(truncate=False)<jupyter_output><empty_output><jupyter_text>**As you can see, we got only `PROBLEM` entities from the text.**  NER VisualizationWe have sparknlp_display library for visualization. This library works with LightPipeline results.<jupyter_code>sample_text = [pubMedDF.limit(3).collect()[i][0] for i in range(3)]
sample_text[1]
ner_lp = LightPipeline(ner_model)
light_result = ner_lp.fullAnnotate(sample_text[1])
from sparknlp_display import NerVisualizer

visualiser = NerVisualizer()

vis = visualiser.display(light_result[0], label_col='ner_chunk', document_col='document', return_html=True)

# Change color of an entity label
#visualiser.set_label_colors({'PROBLEM':'#008080', 'TEST':'#800080', 'TREATMENT':'#808080'})
#visualiser.display(light_result[0], label_col='ner_chunk')

# Set label filter
# vis = visualiser.display(light_result, label_col='ner_chunk', document_col='document',
                   #labels=['PROBLEM','TEST','TREATMENT])
  
displayHTML(vis)<jupyter_output><empty_output><jupyter_text>**There are many NER models for different purposes in Spark NLP. Lets show what if we use `jsl_ner_wip_clinical` model that has about 80 different NER label.**<jupyter_code>jsl_ner = MedicalNerModel.pretrained("jsl_ner_wip_clinical","en","clinical/models")\
        .setInputCols(["sentence","token","embeddings"])\
        .setOutputCol("ner")

jslPipeline = Pipeline(stages=[
        documentAssembler,
        sentenceDetector,
        tokenizer,
        word_embeddings,
        pos_tagger,
        jsl_ner,
        ner_converter])


empty_data = spark.createDataFrame([[""]]).toDF("text")

jsl_model = jslPipeline.fit(empty_data)
jsl_lp = LightPipeline(jsl_model)
jsl_light_result = jsl_lp.fullAnnotate(sample_text[1])
from sparknlp_display import NerVisualizer

visualiser = NerVisualizer()

vis = visualiser.display(jsl_light_result[0], label_col='ner_chunk', document_col='document', return_html=True)

displayHTML(vis)<jupyter_output><empty_output><jupyter_text>**If you want to go over more about NER, you can check this comprehensive notebook :**https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/Certification_Trainings/Healthcare/databricks_notebooks/1.Clinical_Named_Entity_Recognition_Model_v3.0.ipynb  2. Clinical AssertionNow we will check the assertion status of the clinical entities. We will use `ner_clinical_large` model for NER detection, and `assertion_dl` model for checking the assertion status of detected entities. While doing that, we will use the same pipeline that we created fot detecting NER.<jupyter_code># Assertion model trained on i2b2 (sampled from MIMIC) dataset
# coming from sparknlp_jsl.annotator !!
clinical_assertion = AssertionDLModel.pretrained("assertion_dl", "en", "clinical/models") \
    .setInputCols(["sentence", "ner_chunk", "embeddings"]) \
    .setOutputCol("assertion")
    
assertionPipeline = Pipeline(stages=[
    nerPipeline,
    clinical_assertion
    ])

empty_data = spark.createDataFrame([[""]]).toDF("text")

assertion_model = assertionPipeline.fit(empty_data)<jupyter_output><empty_output><jupyter_text>**This time we will use LightPipeline while implementing.**<jupyter_code>sample_text[0]
assertion_light = LightPipeline(assertion_model)<jupyter_output><empty_output><jupyter_text>**We can use `annotate` method to get faster results for short sentences.**<jupyter_code>assertion_anno_res = assertion_light.annotate(sample_text[0])
assertion_anno_res.keys()<jupyter_output><empty_output><jupyter_text>**Lets create a pandas dataframe to see our results obviously.**<jupyter_code>pd.DataFrame(list(zip(assertion_anno_res["ner_chunk"], assertion_anno_res["assertion"])), columns=["ner_chunk", "assertion"])<jupyter_output><empty_output><jupyter_text>**This time we will use `fullAnnotate` method on our text to get metadata results.**<jupyter_code>assertion_result = assertion_light.fullAnnotate(sample_text[0])[0]
assertion_result.keys()
chunks=[]
entities=[]
status=[]

for n,m in zip(assertion_result['ner_chunk'],assertion_result['assertion']):

    chunks.append(n.result)
    entities.append(n.metadata['entity']) 
    status.append(m.result)

df = pd.DataFrame({'chunks':chunks, 'entities':entities, 'assertion':status})
df<jupyter_output><empty_output><jupyter_text>**Also we can filter assertion results by using `AssertionFilterer` annotator. We will use the same pipeline that we vreated before to get the assertions. We will try to get only `present` assertions.**<jupyter_code>assertion_filterer = AssertionFilterer()\
      .setInputCols("sentence","ner_chunk","assertion")\
      .setOutputCol("assertion_filtered")\
      .setWhiteList(["present"])

assertionFilteredPipeline = Pipeline(stages=[
    assertionPipeline,
    assertion_filterer
    ])

empty_data = spark.createDataFrame([[""]]).toDF("text")

assertion_filtered_model = assertionFilteredPipeline.fit(empty_data)
assertion_filtered_light = LightPipeline(assertion_filtered_model)
assertion_filtered_result = assertion_filtered_light.fullAnnotate(sample_text[0])[0]
assertion_filtered_result.keys()
assertion_filtered_result["assertion_filtered"]<jupyter_output><empty_output><jupyter_text>Here is the `present` entities.<jupyter_code>chunks=[]
entities=[]


for n in assertion_filtered_result['assertion_filtered']:

    chunks.append(n.result)
    entities.append(n.metadata['entity']) 


filtered_df = pd.DataFrame({'chunks':chunks, 'entities':entities})

filtered_df<jupyter_output><empty_output><jupyter_text>Assertion VisualizationWe can visualize the assertion status of detected entities by using `AssertionVisualizer` module of `sparknlp_display` library.<jupyter_code>from sparknlp_display import AssertionVisualizer

assertion_vis = AssertionVisualizer()

## To set custom label colors:
assertion_vis.set_label_colors({'TREATMENT':'#008080', 'PROBLEM':'#800080'}) #set label colors by specifying hex codes

vis = assertion_vis.display(assertion_result, 
                            label_col = 'ner_chunk', 
                            assertion_col = 'assertion',
                            document_col = 'document' ,
                            return_html=True
                      )

displayHTML(vis)<jupyter_output><empty_output><jupyter_text>**If you want to go more over about assertion model examples, you can check this comprehensive notebook :**https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/Certification_Trainings/Healthcare/2.Clinical_Assertion_Model.ipynb  3. Relation Extraction In this section, we will show an example of relation extraction models. We will use the same NER pipeline that we created before to extract clinical entities and `re_clinical` model to extract relations between these entities. The set of relations defined in the 2010 i2b2 relation challenge:**TrIP:** A certain treatment has improved or cured a medical problem (eg, ‘infection resolved with antibiotic course’)**TrWP:** A patient's medical problem has deteriorated or worsened because of or in spite of a treatment being administered (eg, ‘the tumor was growing despite the drain’)**TrCP:** A treatment caused a medical problem (eg, ‘penicillin causes a rash’)**TrAP:** A treatment administered for a medical problem (eg, ‘Dexamphetamine for narcolepsy’)**TrNAP:** The administration of a treatment was avoided because of a medical problem (eg, ‘Ralafen which is contra-indicated because of ulcers’)**TeRP:** A test has revealed some medical problem (eg, ‘an echocardiogram revealed a pericardial effusion’)**TeCP:** A test was performed to investigate a medical problem (eg, ‘chest x-ray done to rule out pneumonia’)**PIP:** Two problems are related to each other (eg, ‘Azotemia presumed secondary to sepsis’)<jupyter_code>dependency_parser = DependencyParserModel()\
    .pretrained("dependency_conllu", "en")\
    .setInputCols(["sentence", "pos_tags", "token"])\
    .setOutputCol("dependencies")

clinical_re_Model = RelationExtractionModel()\
    .pretrained("re_clinical", "en", 'clinical/models')\
    .setInputCols(["embeddings", "pos_tags", "ner_chunk", "dependencies"])\
    .setOutputCol("relations")\
    .setMaxSyntacticDistance(4)\
    #.setRelationPairs(["problem-test", "problem-treatment"]) we can set the possible relation pairs (if not set, all the relations will be calculated)

relPipeline = Pipeline(stages=[
    nerPipeline,
    dependency_parser,
    clinical_re_Model
])


empty_data = spark.createDataFrame([[""]]).toDF("text")

rel_model = relPipeline.fit(empty_data)
rel_model.stages
sample_text[1]
rel_light = LightPipeline(rel_model)
relation_res = rel_light.fullAnnotate(sample_text[1])[0]
relation_res.keys()
rel_pairs=[]
  
for rel in relation_res["relations"]:
    rel_pairs.append((
          rel.result, 
          rel.metadata['entity1'], 
          rel.metadata['entity1_begin'],
          rel.metadata['entity1_end'],
          rel.metadata['chunk1'], 
          rel.metadata['entity2'],
          rel.metadata['entity2_begin'],
          rel.metadata['entity2_end'],
          rel.metadata['chunk2'], 
          rel.metadata['confidence']
      ))

rel_df = pd.DataFrame(rel_pairs, columns=['relation','entity1','entity1_begin','entity1_end','chunk1','entity2','entity2_begin','entity2_end','chunk2', 'confidence'])
rel_df[rel_df.relation!="O"]<jupyter_output><empty_output><jupyter_text>Relation VisualizationWe can visualize relations between entities by using `RelationExtractionVisualizer` module of `sparknlp_display` lìbrary.<jupyter_code>from sparknlp_display import RelationExtractionVisualizer

re_vis = RelationExtractionVisualizer()

vis = re_vis.display(relation_res,
                     relation_col = 'relations',
                     document_col = 'document',
                     show_relations=True,
                     return_html=True)

displayHTML(vis)<jupyter_output><empty_output><jupyter_text>**If you want to go more over about relation extraction model examples, you can check this comprehensive notebook :**https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/Certification_Trainings/Healthcare/10.Clinical_Relation_Extraction.ipynb  4. Entity Resolution There are many entity resolution models for different kinds of purposes in Spark NLP. But mainly, we can collect these models in two categories:* Chunk Entity Resolver Models* Sentence Entity Resolver ModelsHere are some of the resolver models in Spark NLP:- sbiobertresolve_icd10cm - sbiobertresolve_icd10cm_augmented- sbiobertresolve_icd10cm_slim_normalized- sbiobertresolve_icd10cm_slim_billable_hcc- sbertresolve_icd10cm_slim_billable_hcc_med- sbiobertresolve_icd10pcs- sbiobertresolve_snomed_findings (with clinical_findings concepts from CT version)- sbiobertresolve_snomed_findings_int  (with clinical_findings concepts from INT version)- sbiobertresolve_snomed_auxConcepts (with Morph Abnormality, Procedure, Substance, Physical Object, Body Structure concepts from CT version)- sbiobertresolve_snomed_auxConcepts_int  (with Morph Abnormality, Procedure, Substance, Physical Object, Body Structure concepts from INT version)- sbiobertresolve_rxnorm- sbiobertresolve_rxcui- sbiobertresolve_icdo- sbiobertresolve_cpt- sbiobertresolve_loinc- sbiobertresolve_HPO- sbiobertresolve_umls_major_concepts- sbiobertresolve_umls_findings- ... We will use the same NER pipeline and `sbiobertresolve_icd10cm_slim_billable_hcc` ICD10 CM entity resolver model.<jupyter_code>c2doc = Chunk2Doc()\
      .setInputCols("ner_chunk")\
      .setOutputCol("ner_chunk_doc") 

sbert_embedder = BertSentenceEmbeddings\
      .pretrained("sbert_jsl_medium_uncased",'en','clinical/models')\
      .setInputCols(["ner_chunk_doc"])\
      .setOutputCol("sbert_embeddings")

icd_resolver = SentenceEntityResolverModel.pretrained("sbertresolve_icd10cm_slim_billable_hcc_med","en", "clinical/models") \
      .setInputCols(["ner_chunk", "sbert_embeddings"]) \
      .setOutputCol("icd10_code")\
      .setDistanceFunction("EUCLIDEAN")


resolverPipeline = Pipeline(stages=[
        nerPipeline,
        c2doc,
        sbert_embedder,
        icd_resolver
    
])

empty_data = spark.createDataFrame([[""]]).toDF("text")
resolver_model = resolverPipeline.fit(empty_data)
res_light = LightPipeline(resolver_model)
res_anno = res_light.annotate("bladder cancer")
res_anno
list(zip(res_anno["ner_chunk"], res_anno["icd10_code"]))
resolver_res = res_light.fullAnnotate(sample_text[1])[0]
resolver_res.keys()
chunks = []
codes = []
begin = []
end = []
resolutions= []
all_distances = []
all_codes= []
all_cosines = []
all_k_aux_labels= []
confidence = []
entity = []

for chunk, code in zip(resolver_res['ner_chunk'], resolver_res["icd10_code"]):

    begin.append(chunk.begin)
    entity.append(chunk.metadata['entity'])
    end.append(chunk.end)
    chunks.append(chunk.result)
    codes.append(code.result) 
    confidence.append(code.metadata['confidence'])
    all_codes.append(code.metadata['all_k_results'].split(':::'))
    resolutions.append(code.metadata['all_k_resolutions'].split(':::'))
    all_distances.append(code.metadata['all_k_distances'].split(':::'))
    all_cosines.append(code.metadata['all_k_cosine_distances'].split(':::'))
    all_k_aux_labels.append(code.metadata['all_k_aux_labels'].split(':::'))
    
df = pd.DataFrame({'chunks':chunks, 'entity':entity, 'begin': begin, 'end':end, 'code':codes, 'all_codes':all_codes, 
                   'resolutions':resolutions, 'all_k_aux_labels':all_k_aux_labels,'all_distances':all_cosines})



df['billable'] = df['all_k_aux_labels'].apply(lambda x: [i.split('||')[0] for i in x])
df['hcc_status'] = df['all_k_aux_labels'].apply(lambda x: [i.split('||')[1] for i in x])
df['hcc_score'] = df['all_k_aux_labels'].apply(lambda x: [i.split('||')[2] for i in x])
df['confidence'] = confidence

df = df.drop(['all_k_aux_labels'], axis=1)
pd.set_option("display.max_colwidth", 100)
df<jupyter_output><empty_output><jupyter_text>**Lets check the confidence level > 0.5 results**<jupyter_code>df[df.confidence.astype(float) > 0.5]<jupyter_output><empty_output><jupyter_text>Entity Resolution Visualization<jupyter_code>from sparknlp_display import EntityResolverVisualizer

er_vis = EntityResolverVisualizer()


## To set custom label colors:
er_vis.set_label_colors({'TREATMENT':'#800080', 'PROBLEM':'#77b5fe'}) #set label colors by specifying hex codes

vis = er_vis.display(resolver_res, 
                     label_col='ner_chunk', 
                     resolution_col = 'icd10_code',
                     document_col='document',
                     return_html=True)

displayHTML(vis)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12341443596804044, 'fraction_numerical': 0.009253825666952558, 'mean_word_length': 5.177914110429448, 'pattern_counts': {'":': 0, '<': 81, '<?xml version=': 0, '>': 96, 'https://': 8, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '639717', 'license': 'Apache-2.0', 'n_tokens_mistral': 7360, 'n_tokens_neox': 6944, 'n_words': 1432, 'path': 'databricks/python/healthcare_case_studies/Get_Started_Spark_NLP_for_Healthcare.ipynb', 'repo_name': 'hatrungduc/spark-nlp-workshop', 'size': '22153'}