<jupyter_start><jupyter_text>![JohnSnowLabs](https://nlp.johnsnowlabs.com/assets/images/logo.png)[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/streamlit_notebooks/healthcare/DEID_EHR_DATA.ipynb)  **De-identify Structured Data** To run this yourself, you will need to upload your license keys to the notebook. Just Run The Cell Below in order to do that. Also You can open the file explorer on the left side of the screen and upload `license_keys.json` to the folder that opens.Otherwise, you can look at the example outputs at the bottom of the notebook.  1. Colab Setup Import license keys<jupyter_code>import os
import json

from google.colab import files

license_keys = files.upload()

with open(list(license_keys.keys())[0]) as f:
    license_keys = json.load(f)

sparknlp_version = license_keys["PUBLIC_VERSION"]
jsl_version = license_keys["JSL_VERSION"]

print ('SparkNLP Version:', sparknlp_version)
print ('SparkNLP-JSL Version:', jsl_version)<jupyter_output><empty_output><jupyter_text>Install dependencies<jupyter_code>%%capture
for k,v in license_keys.items(): 
    %set_env $k=$v

!wget https://raw.githubusercontent.com/JohnSnowLabs/spark-nlp-workshop/master/jsl_colab_setup.sh
!bash jsl_colab_setup.sh<jupyter_output><empty_output><jupyter_text>Import dependencies into Python<jupyter_code>import pandas as pd
from pyspark.ml import Pipeline
from pyspark.sql import SparkSession
import pyspark.sql.functions as F
from tabulate import tabulate
import sparknlp
from sparknlp.annotator import *
from sparknlp_jsl.annotator import *
from sparknlp.base import *
import sparknlp_jsl<jupyter_output><empty_output><jupyter_text>Start the Spark session<jupyter_code>spark = sparknlp_jsl.start(license_keys['SECRET'])

# manually start session
# params = {"spark.driver.memory" : "16G",
#           "spark.kryoserializer.buffer.max" : "2000M",
#           "spark.driver.maxResultSize" : "2000M"}

# spark = sparknlp_jsl.start(license_keys['SECRET'],params=params)<jupyter_output><empty_output><jupyter_text>2. Download Structured PHI Data and Create a `DataFrame`<jupyter_code>!wget -q https://raw.githubusercontent.com/JohnSnowLabs/spark-nlp-workshop/master/tutorials/Certification_Trainings/Healthcare/data/hipaa-table-001.txt
df = spark.read.format("csv") \
    .option("sep", "\t") \
    .option("inferSchema", "true") \
    .option("header", "true") \
    .load("hipaa-table-001.txt")

df = df.withColumnRenamed("PATIENT","NAME")
df.show(truncate=False)<jupyter_output>+---------------+----------+---+----------------------------------------------------+-------+--------------+---+---+
|NAME           |DOB       |AGE|ADDRESS                                             |ZIPCODE|TEL           |SBP|DBP|
+---------------+----------+---+----------------------------------------------------+-------+--------------+---+---+
|<NAME>|04/02/1935|83 |711-2880 Nulla St. Mankato Mississippi              |69200  |(257) 563-7401|101|42 |
|<NAME>    |03/10/2009|9  |P.O. Box 283 8562 Fusce Rd. <NAME>      |20620  |(372) 587-2335|159|122|
|<NAME>    |11/01/1921|98 |5543 Aliquet St. Fort Dodge GA                      |20783  |(717) 450-4729|149|52 |
|<NAME>  |13/02/2002|16 |Ap #867-859 Sit Rd. Azusa New York                  |39531  |(793) 151-6230|134|115|
|<NAME>   |20/08/1942|76 |7292 Dictum Av. San Antonio MI                      |47096  |(492) 709-6392|139|78 |
|<NAME>     |12/05/1973|45 |Ap #651-8679 Sodales Av. Tamunin[...]<jupyter_text>3. De-identify using Obfuscation Method<jupyter_code>from sparknlp_jsl.structured_deidentification import StructuredDeidentification<jupyter_output><empty_output><jupyter_text>We will obfuscate `NAME` column as `PATIENT`, `AGE` column as `AGE` and `TEL` column as `PHONE`.<jupyter_code>obfuscator = StructuredDeidentification(spark,{"NAME" : "PATIENT", "AGE" : "AGE", "TEL" : "PHONE"}, obfuscateRefSource='faker')
obfuscator_df = obfuscator.obfuscateColumns(df)
obfuscator_df.select("NAME", "AGE", "TEL").show(truncate=False)<jupyter_output>+------------------+----+-----------------+
|NAME              |AGE |TEL              |
+------------------+----+-----------------+
|[<NAME>] |[81]|[736 990 188]    |
|[<NAME>]      |[40]|[21 258 717 9805]|
|[<NAME>]   |[65]|[659 625 303]    |
|[<NAME>]    |[3] |[(07) 4550 6193] |
|[<NAME>] |[42]|[488 73 877]     |
|[<NAME>]  |[81]|[078 4260 1147]  |
|[<NAME>air]     |[69]|[412 5473]       |
|[<NAME>]   |[75]|[21 226 725 6159]|
|[<NAME>]  |[53]|[97 895809]      |
|[<NAME>]   |[16]|[22 525335]      |
|[<NAME>]      |[66]|[97 895809]      |
|[<NAME>]  |[36]|[085 728 5749]   |
|[<NAME>]   |[40]|[(02) 6710 7447] |
|[<NAME>]       |[65]|[0359 1465851]   |
|[<NAME>]      |[76]|[018-8430091]    |
|[<NAME>]|[75]|[08273 74 50 77] |
|[<NAME>]    |[47]|[722 778 902]    |
|[<NAME>]    |[35]|[734 605 387]    |
|[<NAME>]  |[62]|[052 988 99 51]  |
|[<NAME>]  |[30]|[95 117[...]<jupyter_text>The annotator does not have fake `DATE` chunks by default. Let's do it manually. We can create a `faker` dictionary for `DOB` column as `DATE` label then we obfuscate `DOB` column as well.<jupyter_code>obfuscator_unique_ref_test = '''2022-11-1#DATE
2033-10-30#DATE
2011-8-22#DATE
2005-11-1#DATE
2008-10-30#DATE
2044-8-22#DATE
2022-04-1#DATE
2033-05-30#DATE
2011-09-22#DATE
2005-12-1#DATE
2008-02-30#DATE
2044-03-22#DATE
2055-11-1#DATE
2066-10-30#DATE
2077-8-22#DATE
2088-11-1#DATE
2099-10-30#DATE
2100-8-22#DATE
2111-04-1#DATE
2122-05-30#DATE
2133-09-22#DATE
2144-12-1#DATE
2155-02-30#DATE
2166-03-22#DATE'''

with open('obfuscator_unique_ref_test.txt', 'w') as f:
  f.write(obfuscator_unique_ref_test)
obfuscator = StructuredDeidentification(spark,{"NAME":"PATIENT","AGE":"AGE", "DOB":"DATE", "TEL":"PHONE"}, obfuscateRefFile="/content/obfuscator_unique_ref_test.txt")
obfuscator_df = obfuscator.obfuscateColumns(df)

obfuscator_df.select("NAME", "DOB", "AGE", "TEL").show(truncate=False)<jupyter_output>+------------------+------------+----+----------------+
|NAME              |DOB         |AGE |TEL             |
+------------------+------------+----+----------------+
|[<NAME>]   |[2099-10-30]|[24]|[0473 75 74 88] |
|[Theron Blend]    |[2111-04-1] |[73]|[814-297-1404]  |
|[<NAME>]      |[2099-10-30]|[62]|[699 792 243]   |
|[<NAME>]       |[2066-10-30]|[63]|[0356 7386222]  |
|[<NAME>]      |[2005-11-1] |[55]|[09704 13 48 83]|
|[<NAME>]   |[2044-03-22]|[35]|[78 577 036]    |
|[<NAME>]    |[2005-12-1] |[6] |[444 14 907]    |
|[<NAME>]    |[2122-05-30]|[76]|[30-88-20-94]   |
|[<NAME>]     |[2133-09-22]|[23]|[448 8003]      |
|[<NAME>]      |[2033-10-30]|[39]|[24 102884]     |
|[<NAME>] |[2022-04-1] |[56]|[250-293-9941]  |
|[<NAME>]  |[2144-12-1] |[59]|[0613-9040212]  |
|[<NAME>] |[2005-12-1] |[73]|[(02) 6765 9044]|
|[<NAME>] |[2088-11-1] |[62]|[95 698307]     |
|[<NAME>]      |[2166-03-22]|[76]|[(022) 375[...]
{'fraction_non_alphanumeric': 0.2352092352092352, 'fraction_numerical': 0.12972582972582972, 'mean_word_length': 3.1753012048192772, 'pattern_counts': {'":': 4, '<': 71, '<?xml version=': 0, '>': 71, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '659654', 'license': 'Apache-2.0', 'n_tokens_mistral': 3349, 'n_tokens_neox': 2745, 'n_words': 498, 'path': 'tutorials/streamlit_notebooks/healthcare/DEID_EHR_DATA.ipynb', 'repo_name': 'hatrungduc/spark-nlp-workshop', 'size': '7194'}