<jupyter_start><jupyter_text>Procedure Occurrence Table MappingThis is an attempt at mapping FHIR to OMOP using the following guide: https://build.fhir.org/ig/HL7/cdmh/profiles.htmlomop-to-fhir-mappingsIn this notebook we are mapping FHIR to the OMOP Procedure Occurrence Table  Load Data Frame from Parquet Catalog File<jupyter_code>from pyspark.sql import SparkSession
from pyspark.sql.functions import dayofmonth,month,year,to_date,trunc,split,explode,array,col

# Create a local Spark session
spark = SparkSession.builder.appName('etl').getOrCreate()
# Reads file 
df = spark.read.parquet('data/catalog.parquet')<jupyter_output><empty_output><jupyter_text>Data Frame schema<jupyter_code>#df.printSchema()<jupyter_output><empty_output><jupyter_text>Procedure Mapping  Filter By Procedure Resource type<jupyter_code>filtered = df.filter(df['resourceType'] == 'Procedure')
#filtered.printSchema()<jupyter_output><empty_output><jupyter_text>Selects relevant fields. Using alias so we don't have columns with the same name<jupyter_code>Procedure = filtered.select("id",
  col("encounter.reference").alias("visit_occurrence_id"),
  col("subject.reference").alias("person_id"),
  "code.coding",
  "performedPeriod",
  "performer",
  "extension")<jupyter_output><empty_output><jupyter_text>Extract the start date and time from the period field.<jupyter_code>#splits the date and time
split_start = split(Procedure['performedPeriod.start'], 'T')

#assigns each to a column 
procedure_date_and_time = Procedure\
    .withColumn("procedure_date",split_start.getItem(0))\
    .withColumn("procedure_datetime",split_start.getItem(1))<jupyter_output><empty_output><jupyter_text>Drop columns no longer needed<jupyter_code>dropped  = procedure_date_and_time.drop("performedPeriod")<jupyter_output><empty_output><jupyter_text>Rename the columns<jupyter_code>procedure_occurnace = dropped\
    .withColumnRenamed("id","procedure_occurrence_id")\
    .withColumnRenamed("coding","procedure_concept_id")\
    .withColumnRenamed("extension","procedure_type_concept_id")\
    .withColumnRenamed("extension","quantity")\
    .withColumnRenamed("performer","provider_id")<jupyter_output><empty_output><jupyter_text>Shows mapped output table<jupyter_code>procedure_occurnace.show(5)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11567001309471846, 'fraction_numerical': 0.001745962461807071, 'mean_word_length': 8.711864406779661, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '322170', 'license': 'MIT', 'n_tokens_mistral': 784, 'n_tokens_neox': 747, 'n_words': 126, 'path': 'notebooks/PROCEDURE_OCCURRENCE.ipynb', 'repo_name': 'spe-uob/HealthcareLakeETL', 'size': '2291'}