<jupyter_start><jupyter_text>![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/streamlit_notebooks/healthcare/NER_DIAG_PROC.ipynb)  **Detect diagnosis and procedures** To run this yourself, you will need to upload your license keys to the notebook. Just Run The Cell Below in order to do that. Also You can open the file explorer on the left side of the screen and upload `license_keys.json` to the folder that opens.Otherwise, you can look at the example outputs at the bottom of the notebook.  1. Colab Setup Import license keys<jupyter_code>import os
import json

from google.colab import files

license_keys = files.upload()

with open(list(license_keys.keys())[0]) as f:
    license_keys = json.load(f)

sparknlp_version = license_keys["PUBLIC_VERSION"]
jsl_version = license_keys["JSL_VERSION"]

print ('SparkNLP Version:', sparknlp_version)
print ('SparkNLP-JSL Version:', jsl_version)<jupyter_output><empty_output><jupyter_text>Install dependencies<jupyter_code>%%capture
for k,v in license_keys.items(): 
    %set_env $k=$v

!wget https://raw.githubusercontent.com/JohnSnowLabs/spark-nlp-workshop/master/jsl_colab_setup.sh
!bash jsl_colab_setup.sh

# Install Spark NLP Display for visualization
!pip install --ignore-installed spark-nlp-display<jupyter_output><empty_output><jupyter_text>Import dependencies into Python and start the Spark session<jupyter_code>import pandas as pd
from pyspark.ml import Pipeline
from pyspark.sql import SparkSession
import pyspark.sql.functions as F

import sparknlp
from sparknlp.annotator import *
from sparknlp_jsl.annotator import *
from sparknlp.base import *
import sparknlp_jsl

spark = sparknlp_jsl.start(license_keys['SECRET'])

# manually start session
# params = {"spark.driver.memory" : "16G",
#           "spark.kryoserializer.buffer.max" : "2000M",
#           "spark.driver.maxResultSize" : "2000M"}

# spark = sparknlp_jsl.start(license_keys['SECRET'],params=params)<jupyter_output><empty_output><jupyter_text>2. Select the NER model and construct the pipeline Select the NER model - Diagnosis & Procedures models: **ner_diseases, ner_clinical, ner_jsl**For more details: https://github.com/JohnSnowLabs/spark-nlp-modelspretrained-models---spark-nlp-for-healthcare<jupyter_code># You can change this to the model you want to use and re-run cells below.
# Diagnosis & Procedures models: ner_diseases, ner_clinical, ner_jsl
MODEL_NAME = "ner_diseases"<jupyter_output><empty_output><jupyter_text>Create the pipeline<jupyter_code>document_assembler = DocumentAssembler() \
    .setInputCol('text')\
    .setOutputCol('document')

sentence_detector = SentenceDetector() \
    .setInputCols(['document'])\
    .setOutputCol('sentence')

tokenizer = Tokenizer()\
    .setInputCols(['sentence']) \
    .setOutputCol('token')

word_embeddings = WordEmbeddingsModel.pretrained('embeddings_clinical', 'en', 'clinical/models') \
    .setInputCols(['sentence', 'token']) \
    .setOutputCol('embeddings')

clinical_ner = MedicalNerModel.pretrained(MODEL_NAME, "en", "clinical/models") \
.setInputCols(["sentence", "token", "embeddings"])\
.setOutputCol("ner")

ner_converter = NerConverter()\
    .setInputCols(['sentence', 'token', 'ner']) \
    .setOutputCol('ner_chunk')

nlp_pipeline = Pipeline(stages=[
    document_assembler, 
    sentence_detector,
    tokenizer,
    word_embeddings,
    clinical_ner,
    ner_converter])<jupyter_output>embeddings_clinical download started this may take some time.
Approximate size to download 1.6 GB
[OK!]
ner_diseases download started this may take some time.
Approximate size to download 13.8 MB
[OK!]<jupyter_text>3. Create example inputs<jupyter_code># Enter examples as strings in this array
input_list = [
    """FINDINGS: The patient was found upon excision of the cyst that it contained a large Prolene suture, which is multiply knotted as it always is; beneath this was a very small incisional hernia, the hernia cavity, which contained omentum; the hernia was easily repaired.

DESCRIPTION OF PROCEDURE: The patient was identified, then taken into the operating room, where after induction of an LMA anesthetic, his abdomen was prepped with Betadine solution and draped in sterile fashion. The puncta of the wound lesion was infiltrated with methylene blue and peroxide. The lesion was excised and the existing scar was excised using an ellipse and using a tenotomy scissors, the cyst was excised down to its base. In doing so, we identified a large Prolene suture within the wound and followed this cyst down to its base at which time we found that it contained omentum and was in fact overlying a small incisional hernia. The cyst was removed in its entirety, divided from the omentum using a Metzenbaum and tying with 2-0 silk ties. The hernia repair was undertaken with interrupted 0 Vicryl suture with simple sutures. The wound was then irrigated and closed with 3-0 Vicryl subcutaneous and 4-0 Vicryl subcuticular and Steri-Strips. Patient tolerated the procedure well. Dressings were applied and he was taken to recovery room in stable condition. """
]<jupyter_output><empty_output><jupyter_text>4. Use the pipeline to create outputs<jupyter_code>empty_df = spark.createDataFrame([['']]).toDF('text')
pipeline_model = nlp_pipeline.fit(empty_df)
df = spark.createDataFrame(pd.DataFrame({'text': input_list}))
result = pipeline_model.transform(df)<jupyter_output><empty_output><jupyter_text>5. Visualize results<jupyter_code>from sparknlp_display import NerVisualizer

NerVisualizer().display(
    result = result.collect()[0],
    label_col = 'ner_chunk',
    document_col = 'document'
)<jupyter_output><empty_output><jupyter_text>Visualize outputs as data frame<jupyter_code>exploded = F.explode(F.arrays_zip('ner_chunk.result', 'ner_chunk.metadata'))
select_expression_0 = F.expr("cols['0']").alias("chunk")
select_expression_1 = F.expr("cols['1']['entity']").alias("ner_label")
result.select(exploded.alias("cols")) \
    .select(select_expression_0, select_expression_1).show(truncate=False)
result = result.toPandas()<jupyter_output>+------------------------------+---------+
|chunk                         |ner_label|
+------------------------------+---------+
|the cyst                      |Disease  |
|a large Prolene suture        |Disease  |
|a very small incisional hernia|Disease  |
|the hernia cavity             |Disease  |
|omentum                       |Disease  |
|the hernia                    |Disease  |
|the wound lesion              |Disease  |
|The lesion                    |Disease  |
|the existing scar             |Disease  |
|the cyst                      |Disease  |
|the wound                     |Disease  |
|this cyst down to its base    |Disease  |
|a small incisional hernia     |Disease  |
|The cyst                      |Disease  |
|The wound                     |Disease  |
+------------------------------+---------+
{'fraction_non_alphanumeric': 0.11320754716981132, 'fraction_numerical': 0.004965243296921549, 'mean_word_length': 4.172413793103448, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '17387', 'license': 'Apache-2.0', 'n_tokens_mistral': 2272, 'n_tokens_neox': 2112, 'n_words': 620, 'path': 'tutorials/streamlit_notebooks/healthcare/NER_DIAG_PROC.ipynb', 'repo_name': 'hatrungduc/spark-nlp-workshop', 'size': '7049'}