<jupyter_start><jupyter_text>![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/streamlit_notebooks/healthcare/NER_CHEMPROT_CLINICAL.ipynb)  **Detect chemical compounds and genes** To run this yourself, you will need to upload your license keys to the notebook. Just Run The Cell Below in order to do that. Also You can open the file explorer on the left side of the screen and upload `license_keys.json` to the folder that opens.Otherwise, you can look at the example outputs at the bottom of the notebook.  1. Colab Setup Import license keys<jupyter_code>import json
import os

from google.colab import files

license_keys = files.upload()

with open(list(license_keys.keys())[0]) as f:
    license_keys = json.load(f)

# Defining license key-value pairs as local variables
locals().update(license_keys)

# Adding license key-value pairs to environment variables
os.environ.update(license_keys)<jupyter_output><empty_output><jupyter_text>Install dependencies<jupyter_code># Installing pyspark and spark-nlp
! pip install --upgrade -q pyspark==3.1.2 spark-nlp==$PUBLIC_VERSION

# Installing Spark NLP Healthcare
! pip install --upgrade -q spark-nlp-jsl==$JSL_VERSION  --extra-index-url https://pypi.johnsnowlabs.com/$SECRET

# Installing Spark NLP Display Library for visualization
! pip install -q spark-nlp-display<jupyter_output><empty_output><jupyter_text>Import dependencies into Python and start the Spark session<jupyter_code>import pandas as pd
from pyspark.ml import Pipeline
from pyspark.sql import SparkSession
import pyspark.sql.functions as F

import sparknlp
from sparknlp.annotator import *
from sparknlp_jsl.annotator import *
from sparknlp.base import *
import sparknlp_jsl

spark = sparknlp_jsl.start(license_keys['SECRET'])

# manually start session
'''
builder = SparkSession.builder \
    .appName('Spark NLP Licensed') \
    .master('local[*]') \
    .config('spark.driver.memory', '16G') \
    .config('spark.serializer', 'org.apache.spark.serializer.KryoSerializer') \
    .config('spark.kryoserializer.buffer.max', '2000M') \
    .config('spark.jars.packages', 'com.johnsnowlabs.nlp:spark-nlp_2.11:' +sparknlp.version()) \
    .config('spark.jars', f'https://pypi.johnsnowlabs.com/{secret}/spark-nlp-jsl-{jsl_version}.jar')
'''<jupyter_output><empty_output><jupyter_text>2. Select the NER model and construct the pipeline Select the NER modelFor more details: https://github.com/JohnSnowLabs/spark-nlp-modelspretrained-models---spark-nlp-for-healthcare<jupyter_code>MODEL_NAME = "ner_chemprot_clinical"<jupyter_output><empty_output><jupyter_text>Create the pipeline<jupyter_code>document_assembler = DocumentAssembler() \
    .setInputCol('text')\
    .setOutputCol('document')

sentence_detector = SentenceDetector() \
    .setInputCols(['document'])\
    .setOutputCol('sentence')

tokenizer = Tokenizer()\
    .setInputCols(['sentence']) \
    .setOutputCol('token')

word_embeddings = WordEmbeddingsModel.pretrained('embeddings_clinical', 'en', 'clinical/models') \
    .setInputCols(['sentence', 'token']) \
    .setOutputCol('embeddings')

clinical_ner = MedicalNerModel.pretrained(MODEL_NAME, "en", "clinical/models") \
.setInputCols(["sentence", "token", "embeddings"])\
.setOutputCol("ner")

ner_converter = NerConverter()\
    .setInputCols(['sentence', 'token', 'ner']) \
    .setOutputCol('ner_chunk')

nlp_pipeline = Pipeline(stages=[
    document_assembler, 
    sentence_detector,
    tokenizer,
    word_embeddings,
    clinical_ner,
    ner_converter])<jupyter_output>embeddings_clinical download started this may take some time.
Approximate size to download 1.6 GB
[OK!]
ner_chemprot_clinical download started this may take some time.
Approximate size to download 15.3 MB
[OK!]<jupyter_text>3. Create example inputs<jupyter_code># Enter examples as strings in this array
input_list = [
    """Emerging role of epidermal growth factor receptor inhibition in therapy for advanced malignancy: focus on NSCLC. Combination chemotherapy regimens have emerged as the standard approach in advanced non-small-cell lung cancer. Meta-analyses have demonstrated a 2-month increase in median survival after platinum-based therapy vs. best supportive care, and an absolute 10% improvement in the 1-year survival rate. Just as importantly, cytotoxic therapy has produced benefits in symptom control and quality of life. Newer agents, including the taxanes, vinorelbine, gemcitabine, and irinotecan, have expanded our therapeutic options in the treatment of advanced non-small-cell lung cancer. Despite their contributions, we have reached a therapeutic plateau, with response rates seldom exceeding 30-40% in cooperative group studies and 1-year survival rates stable between 30% and 40%. It is doubtful that substituting one agent for another in various combinations will lead to any further improvement in these rates. The thrust of current research has focused on targeted therapy, and epidermal growth factor receptor inhibition is one of the most promising clinical strategies. Epidermal growth factor receptor inhibitors currently under investigation include the small molecules gefitinib (Iressa, ZD1839) and erlotinib (Tarceva, OSI-774), as well as monoclonal antibodies such as cetuximab (IMC-225, Erbitux). Agents that have only begun to undergo clinical evaluation include CI-1033, an irreversible pan-erbB tyrosine kinase inhibitor, and PKI166 and GW572016, both examples of dual kinase inhibitors (inhibiting epidermal growth factor receptor and Her2). Preclinical models have demonstrated synergy for all these agents in combination with either chemotherapy or radiotherapy, leading to great enthusiasm regarding their ultimate contribution to lung cancer therapy. However, serious clinical challenges persist. These include the identification of the optimal dose(s); the proper integration of these agents into popular, established cytotoxic regimens; and the selection of the optimal setting(s) in which to test these compounds. Both gefitinib and erlotinib have shown clinical activity in pretreated, advanced non-small-cell lung cancer, but placebo-controlled randomized Phase III studies evaluating gefitinib in combination with standard cytotoxic therapy, to our chagrin, have failed to demonstrate a survival advantage compared with chemotherapy alone.""",
]<jupyter_output><empty_output><jupyter_text>4. Use the pipeline to create outputs<jupyter_code>empty_df = spark.createDataFrame([['']]).toDF('text')
pipeline_model = nlp_pipeline.fit(empty_df)
df = spark.createDataFrame(pd.DataFrame({'text': input_list}))
result = pipeline_model.transform(df)<jupyter_output><empty_output><jupyter_text>5. Visualize results<jupyter_code>from sparknlp_display import NerVisualizer

NerVisualizer().display(
    result = result.collect()[0],
    label_col = 'ner_chunk',
    document_col = 'document'
)<jupyter_output><empty_output><jupyter_text>Visualize outputs as data frame<jupyter_code>exploded = F.explode(F.arrays_zip('ner_chunk.result', 'ner_chunk.metadata'))
select_expression_0 = F.expr("cols['0']").alias("chunk")
select_expression_1 = F.expr("cols['1']['entity']").alias("ner_label")
result.select(exploded.alias("cols")) \
    .select(select_expression_0, select_expression_1).show(truncate=False)
result = result.toPandas()<jupyter_output>+--------------------------------+---------+
|chunk                           |ner_label|
+--------------------------------+---------+
|epidermal growth factor receptor|GENE-Y   |
|taxanes                         |CHEMICAL |
|vinorelbine                     |CHEMICAL |
|gemcitabine                     |CHEMICAL |
|irinotecan                      |CHEMICAL |
|epidermal growth factor receptor|GENE-Y   |
|Epidermal growth factor receptor|GENE-Y   |
|gefitinib                       |CHEMICAL |
|Iressa                          |CHEMICAL |
|ZD1839                          |CHEMICAL |
|erlotinib                       |CHEMICAL |
|Tarceva                         |CHEMICAL |
|OSI-774                         |CHEMICAL |
|cetuximab                       |CHEMICAL |
|IMC-225                         |CHEMICAL |
|Erbitux                         |CHEMICAL |
|CI-1033                         |CHEMICAL |
|tyrosine kinase                 |GENE-N   |
|PKI166                          |CHEMICAL |
|GW572016 [...]
{'fraction_non_alphanumeric': 0.10357016613644397, 'fraction_numerical': 0.010604453870625663, 'mean_word_length': 4.150485436893204, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '180091', 'license': 'Apache-2.0', 'n_tokens_mistral': 2644, 'n_tokens_neox': 2388, 'n_words': 722, 'path': 'tutorials/streamlit_notebooks/healthcare/NER_CHEMPROT_CLINICAL.ipynb', 'repo_name': 'fcivardi/spark-nlp-workshop', 'size': '8487'}