<jupyter_start><jupyter_text>![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/streamlit_notebooks/healthcare/ER_ICD10_CM.ipynb)  **ICD10-CM coding** To run this yourself, you will need to upload your license keys to the notebook. Otherwise, you can look at the example outputs at the bottom of the notebook. To upload license keys, open the file explorer on the left side of the screen and upload `workshop_license_keys.json` to the folder that opens.  1. Colab Setup Import license keys<jupyter_code>import os
import json

with open('/content/spark_nlp_for_healthcare.json', 'r') as f:
    license_keys = json.load(f)

license_keys.keys()

secret = license_keys['SECRET']
os.environ['SPARK_NLP_LICENSE'] = license_keys['SPARK_NLP_LICENSE']
os.environ['AWS_ACCESS_KEY_ID'] = license_keys['AWS_ACCESS_KEY_ID']
os.environ['AWS_SECRET_ACCESS_KEY'] = license_keys['AWS_SECRET_ACCESS_KEY']
sparknlp_version = license_keys["PUBLIC_VERSION"]
jsl_version = license_keys["JSL_VERSION"]

print ('SparkNLP Version:', sparknlp_version)
print ('SparkNLP-JSL Version:', jsl_version)<jupyter_output>SparkNLP Version: 2.6.0
SparkNLP-JSL Version: 2.6.0<jupyter_text>Install dependencies<jupyter_code># Install Java
! apt-get update -qq
! apt-get install -y openjdk-8-jdk-headless -qq > /dev/null
! java -version

# Install pyspark
! pip install --ignore-installed -q pyspark==2.4.4

# Install Spark NLP
! pip install --ignore-installed spark-nlp==$sparknlp_version
! python -m pip install --upgrade spark-nlp-jsl==$jsl_version --extra-index-url https://pypi.johnsnowlabs.com/$secret<jupyter_output>openjdk version "11.0.8" 2020-07-14
OpenJDK Runtime Environment (build 11.0.8+10-post-Ubuntu-0ubuntu118.04.1)
OpenJDK 64-Bit Server VM (build 11.0.8+10-post-Ubuntu-0ubuntu118.04.1, mixed mode, sharing)
[K     |████████████████████████████████| 215.7MB 55kB/s 
[K     |████████████████████████████████| 204kB 46.9MB/s 
[?25h  Building wheel for pyspark (setup.py) ... [?25l[?25hdone
Collecting spark-nlp==2.6.0
[?25l  Downloading https://files.pythonhosted.org/packages/e4/30/1bd0abcc97caed518efe527b9146897255dffcf71c4708586a82ea9eb29a/spark_nlp-2.6.0-py2.py3-none-any.whl (125kB)
[K     |████████████████████████████████| 133kB 2.9MB/s 
[?25hInstalling collected packages: spark-nlp
Successfully installed spark-nlp-2.6.0
Looking in indexes: https://pypi.org/simple, https://pypi.johnsnowlabs.com/2.6.0-8388813d58b67fa25bf9cf603393363af96dba16
Collecting spark-nlp-jsl==2.6.0
Requirement already satisfied, skipping upgrade: spark-nlp==2.6.0 in /usr/local/lib/python3.6/dist-packages (from s[...]<jupyter_text>Import dependencies into Python<jupyter_code>os.environ['JAVA_HOME'] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ['PATH'] = os.environ['JAVA_HOME'] + "/bin:" + os.environ['PATH']

import pandas as pd
from pyspark.ml import Pipeline
from pyspark.sql import SparkSession
import pyspark.sql.functions as F

import sparknlp
from sparknlp.annotator import *
from sparknlp_jsl.annotator import *
from sparknlp.base import *
import sparknlp_jsl<jupyter_output><empty_output><jupyter_text>Start the Spark session<jupyter_code>spark = sparknlp_jsl.start(secret)<jupyter_output><empty_output><jupyter_text>2. Select the Entity Resolver model and construct the pipeline **NOTE: The mapping below is an example of how ICD10 resolvers work with different NER models. You can choose different combinations  according to your input data and requirements.**  Select the models:**ICD10 Entity Resolver models:**1.   **chunkresolve_icd10cm_clinical**2.   **chunkresolve_icd10cm_diseases_clinical**3.   **chunkresolve_icd10cm_injuries_clinical**4.   **chunkresolve_icd10cm_musculoskeletal_clinical**5.   **chunkresolve_icd10cm_neoplasms_clinical**6.   **chunkresolve_icd10cm_puerile_clinical**For more details: https://github.com/JohnSnowLabs/spark-nlp-modelspretrained-models---spark-nlp-for-healthcare<jupyter_code>#ner and entity resolver mapping
ner_er_dict = {'chunkresolve_icd10cm_clinical': 'ner_clinical',
              'chunkresolve_icd10cm_diseases_clinical': 'ner_diseases',
              'chunkresolve_icd10cm_injuries_clinical': 'ner_clinical',
              'chunkresolve_icd10cm_musculoskeletal_clinical': 'ner_clinical',
              'chunkresolve_icd10cm_neoplasms_clinical': 'ner_bionlp',
              'chunkresolve_icd10cm_puerile_clinical': 'ner_jsl'}
# ER models are specfic to the codes they are trained on, so we need to filter out entities that will cause noise.
wl_er_dict = {'chunkresolve_icd10cm_clinical': ['PROBLEM'],
              'chunkresolve_icd10cm_diseases_clinical': ['Disease'],
              'chunkresolve_icd10cm_injuries_clinical': ['PROBLEM'],
              'chunkresolve_icd10cm_musculoskeletal_clinical': ['PROBLEM'],
              'chunkresolve_icd10cm_neoplasms_clinical': ['CANCER','PATHOLOGICAL_FORMATION'],
              'chunkresolve_icd10cm_puerile_clinical': ['PROBLEM']}

# Change this to the model you want to use and re-run the cells below.
model = 'chunkresolve_icd10cm_clinical'<jupyter_output><empty_output><jupyter_text>Create the pipeline<jupyter_code>document_assembler = DocumentAssembler() \
    .setInputCol('text')\
    .setOutputCol('document')

sentence_detector = SentenceDetector() \
    .setInputCols(['document'])\
    .setOutputCol('sentences')

tokenizer = Tokenizer()\
    .setInputCols(['sentences']) \
    .setOutputCol('tokens')

embeddings = WordEmbeddingsModel.pretrained('embeddings_clinical', 'en', 'clinical/models')\
    .setInputCols(["sentences", "tokens"])\
    .setOutputCol("embeddings")

ner_model = NerDLModel().pretrained(ner_er_dict[model], 'en', 'clinical/models')\
    .setInputCols("sentences", "tokens", "embeddings")\
    .setOutputCol("ner_tags")   

#using defined whitelist. You can define your own as well.
ner_chunker = NerConverter()\
    .setInputCols(["sentences", "tokens", "ner_tags"])\
    .setOutputCol("ner_chunk").setWhiteList(wl_er_dict[model])

chunk_embeddings = ChunkEmbeddings()\
    .setInputCols("ner_chunk", "embeddings")\
    .setOutputCol("chunk_embeddings")

entity_resolver = \
    ChunkEntityResolverModel.pretrained(model,"en","clinical/models")\
    .setInputCols("tokens","chunk_embeddings").setOutputCol("resolution")
    
pipeline = Pipeline(stages=[
    document_assembler, 
    sentence_detector,
    tokenizer,
    embeddings,
    ner_model,
    ner_chunker,
    chunk_embeddings,
    entity_resolver])

empty_df = spark.createDataFrame([['']]).toDF("text")
pipeline_model = pipeline.fit(empty_df)

light_pipeline = sparknlp.base.LightPipeline(pipeline_model)<jupyter_output>embeddings_clinical download started this may take some time.
Approximate size to download 1.6 GB
[OK!]
ner_clinical download started this may take some time.
Approximate size to download 13.8 MB
[OK!]
chunkresolve_icd10cm_clinical download started this may take some time.
Approximate size to download 166.3 MB
[OK!]<jupyter_text>3. Create example inputs<jupyter_code># Enter examples as strings in this array
input_list = [
"""The patient is a 5-month-old infant who presented initially on Monday with a cold, cough, and runny nose for 2 days. Mom states she had no fever. Her appetite was good but she was spitting up a lot. She had no difficulty breathing and her cough was described as dry and hacky. At that time, physical exam showed a right TM, which was red. Left TM was okay. She was fairly congested but looked happy and playful. She was started on Amoxil and Aldex and we told to recheck in 2 weeks to recheck her ear. Mom returned to clinic again today because she got much worse overnight. She was having difficulty breathing. She was much more congested and her appetite had decreased significantly today. She also spiked a temperature yesterday of 102.6 and always having trouble sleeping secondary to congestion.""",
             ]<jupyter_output><empty_output><jupyter_text>4. Run the pipeline<jupyter_code>df = spark.createDataFrame(pd.DataFrame({"text": input_list}))
result = pipeline_model.transform(df)
light_result = light_pipeline.fullAnnotate(input_list[0])<jupyter_output><empty_output><jupyter_text>5. Visualize Full Pipeline<jupyter_code>result.select(
    F.explode(
        F.arrays_zip('ner_chunk.result', 
                     'ner_chunk.begin',
                     'ner_chunk.end',
                     'ner_chunk.metadata',
                     'resolution.metadata', 'resolution.result')
    ).alias('cols')
).select(
    F.expr("cols['0']").alias('chunk'),
    F.expr("cols['1']").alias('begin'),
    F.expr("cols['2']").alias('end'),
    F.expr("cols['3']['entity']").alias('entity'),
    F.expr("cols['4']['resolved_text']").alias('icd10_description'),
    F.expr("cols['5']").alias('icd10_code'),
).toPandas()<jupyter_output><empty_output><jupyter_text>Light Pipeline<jupyter_code>light_result[0]['resolution']<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12906784335355764, 'fraction_numerical': 0.028240485383342527, 'mean_word_length': 5.205338809034908, 'pattern_counts': {'":': 1, '<': 38, '<?xml version=': 0, '>': 39, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '358699', 'license': 'Apache-2.0', 'n_tokens_mistral': 3352, 'n_tokens_neox': 3023, 'n_words': 653, 'path': 'tutorials/streamlit_notebooks/healthcare/ER_ICD10_CM.ipynb', 'repo_name': 'alexisfatica/spark-nlp-workshop', 'size': '9065'}