<jupyter_start><jupyter_text>The purpose of this notebook is to show on the one hand the use of the FHIR2datset module and on the other hand the underlying functionalities of this module.  Notebook initialization<jupyter_code>%load_ext autoreload
%autoreload 2
import os
txt = os.getcwd()
txt = txt.split("/")[:-1]
parent_path = "/".join(txt)
os.chdir(parent_path)
import logging
logging.basicConfig(filename="log.txt", level=logging.INFO)
# os.getcwd()<jupyter_output><empty_output><jupyter_text>Necessary imports to use the module classically<jupyter_code>from fhir2dataset import Query, FHIRRules, FHIR2DatasetParser<jupyter_output><empty_output><jupyter_text>Additional imports to show the inner workings of the module<jupyter_code>import json
import pprint
import pandas as pd
from IPython.display import Markdown as md
pp = pprint.PrettyPrinter(indent=1)

from fhir2dataset.graph_tools import join_path
from fhir2dataset.url_builder import URLBuilder
from fhir2dataset.visualization_tools import custom_repr
from fhir2dataset.data_class import show_tree<jupyter_output><empty_output><jupyter_text>Loading the example used for the tutorial We will answer this question with the example stored in the following folder:<jupyter_code>dirname = 'tests/5'
filename_config = 'config.json'
filename_question = 'question.md'
filename_sql_like_query = 'infos_test/sql_like_query.md'

with open(os.path.join(dirname, filename_config)) as json_file:
    config = json.load(json_file)
with open(os.path.join(dirname, filename_question), 'r') as mardown_file:
     question= mardown_file.read()
with open(os.path.join(dirname, filename_sql_like_query), 'r') as mardown_file:
    sql_like_query= mardown_file.read()<jupyter_output><empty_output><jupyter_text>What's this module for?  The module will make it possible to answer the following question knowing that the searched data are stored in a database provided with an API compliant with the FHIR standard. The data corresponding to the query will be returned in a tabular format, ideal for data analysis.<jupyter_code>md(question)<jupyter_output><empty_output><jupyter_text>If you are familiar with SQL queries, the kind of query you would have made would have been as follows:Note: the queries that will be made by this module are not exact SQL queries, they are in a way more flexible to be integrated with other services thanks to the configuration file.<jupyter_code>print(sql_like_query)<jupyter_output>SELECT patient.Patient.name.family,patient.Patient.name.given,patient.Patient.birthDate,patient.Patient.id,practitioner.Practitioner.name.family,practitioner.Practitioner.address.use,practitioner.Practitioner.address.line,practitioner.Practitioner.address.city,practitioner.Practitioner.id,allergy.AllergyIntolerance.code.coding.display,allergy.AllergyIntolerance.id,condition.Condition.code.coding.display,condition.Condition.severity.coding.display,condition.Condition.id

FROM AllergyIntolerance AS allergy
INNER JOIN Patient AS patient ON allergy.patient = patient.id
INNER JOIN Practitioner AS practitioner ON allergy.asserter = practitioner.id
INNER JOIN Condition AS condition ON condition.subject = patient.id

WHERE patient.birthdate = "1976"
AND allergy.code = "http://snomed.info/sct|22749300"
AND practitioner.address-city = "paris"
AND condition.code = "http://snomed.info/sct|386661006"<jupyter_text>How to use this module?  1. First of all, the elements of the query you want to make must be translated into a json configuration file.The configuration file for the example is therefore:<jupyter_code>pp.pprint(config)<jupyter_output>{'from': {'allergy': 'AllergyIntolerance',
          'condition': 'Condition',
          'patient': 'Patient',
          'practitioner': 'Practitioner'},
 'join': {'inner': {'allergy': {'asserter': 'practitioner',
                                'patient': 'patient'},
                    'condition': {'subject': 'patient'}}},
 'select': {'allergy': ['AllergyIntolerance.code.coding.display',
                        'AllergyIntolerance.id'],
            'condition': ['Condition.code.coding.display',
                          'Condition.severity.coding.display',
                          'Condition.id'],
            'patient': ['Patient.name.family',
                        'Patient.name.given',
                        'Patient.birthDate',
                        'Patient.id'],
            'practitioner': ['Practitioner.name.family',
                             'Practitioner.address.use',
                             'Practitioner.address.line',
                             'Practitioner[...]<jupyter_text>It is also possible to transform the SQL like query into a configuration file<jupyter_code>parser = FHIR2DatasetParser()
config_from_parser = parser.parse(sql_like_query)
pp.pprint(config_from_parser)
# READ OUTPUT
alias = list(config['from'].keys())[0]
resource_type = config['from'][alias]
url_fhir_doc = f"https://www.hl7.org/fhir/{resource_type.lower()}.html"
md(
    f"a. The information contained in the **'from'** allows you to specify the **FHIR type of resource** "
    f"associated with each alias.<br><br>"
    f"For example, here the alias **{alias}** is a FHIR resource "
    f"of type **{resource_type}** whose corresponding documentation can be found at the following link: [{url_fhir_doc}]({url_fhir_doc})"
)
# READ OUTPUT
data = {
    'alias linked to a parent-type resource':
    {'search parameter leading to a Reference type data in the json of an occurrence of a FHIR resource of the parent type': 'alias linked to a child-type resource'}}

join_how=list(config['join'].keys())[0]
parent_alias = list(config['join'][join_how].keys())[0]
expression = list(config['join'][join_how][parent_alias].keys())[0]
child_alias = config['join'][join_how][parent_alias][expression]
data_example = { parent_alias: {expression: child_alias}}

url_fhir_doc=f"https://www.hl7.org/fhir/{parent_alias.lower()}.html#resource"
md(
    f"b. The information contained in the **'join'** allows you to specify "
    f"**how the aliases are linked together**.<br><br>"
    f"Joins are of the parent-child type, i.e. in a resource instance of the "
    f"parent type there is a [reference](https://www.hl7.org/fhir/references.html#Reference) to an instance of the child resource type. "
    f"This will result in the following structure in the configuration file:<br><br>"
    f"{data}<br><br>"
    f"Example ([FHIR doc associated]({url_fhir_doc})):<br>"
    f"{data_example}<br><br>"
    f"Note: each alias must at least be linked to another alias."
)
# READ OUTPUT
alias = list(config['where'].keys())[0]
url_fhir_doc=f"https://www.hl7.org/fhir/{alias.lower()}.html#search"

data = {
    'alias': {'search param': {'prefix': 'value'}}
}
data_default = {
    'alias': {'search param': 'value'}
}
md(
    f"c. The information contained in the **'where'** allows you to specify the **conditions "
    f"on the search parameters**.<br><br>"
    f"For example, the search parameters that the alias {alias} can have are specified "
    f"[here]({url_fhir_doc}) in the FHIR doc."
    f"This will result in the following structure in the configuration file:<br>"
    f"{data}<br><br>"
    f"The prefixes that can be used are those specified [here](https://www.hl7.org/fhir/search.html#prefix) in the FHIR doc."
    f"It is also possible to use a simplified structure that assumes that the prefix used is eq:<br>"
    f"{data_default}<br><br>"
)
# READ OUTPUT
alias = None
expressions = None
url_fhir_doc = None
resource = None
for key in config['select'].keys():
    if key != 'count':
        alias = key
        expressions = config['select'][key]
        resource = config['from'][key]
        url_fhir_doc = f"https://www.hl7.org/fhir/{resource.lower()}.html#resource"
        break
md(
    f"d. The information contained in the **'select'** allows you to specify the elements that "
    f"must be present in the output table.<br><br>"
    f"For example, for the instances corresponding to the alias {alias} it will be necessary to "
    f"retrieve the information in the JSON targeted by the search parameters and/or fhirpaths: {expressions}. "
    f"It can be observed that these search parameters and/or fhirpath correspond well to the FHIR specifications of "
    f"the associated [{resource} resource](url_fhir_doc).<br><br>"
    f"Since v0.2.0, it is recommanded to use fhirpath instead of search parameters. Indeed, the potential link between "
    f"2 columns is determined with the fhirpath of the 2 columns. By nature, the search parameter hide a fhirpath which "
    f"can lead to an unwanted resulting dataframe."
)<jupyter_output><empty_output><jupyter_text>2. Initilize an instance of the **Query class** 3 things are needed to initialize this instance:* fhir_api_url: the Service Base URL (e.g. [http://hapi.fhir.org/baseR4/](http://hapi.fhir.org/baseR4/))* a FHIRRules instance containing information about the search parameters of the targeted FHIR API * config : a json file in the previously described format<jupyter_code>fhir_api_url = 'http://hapi.fhir.org/baseR4/'

#fhir_rules can be instantiated only once and then used for all the querries
fhir_rules=FHIRRules(fhir_api_url=fhir_api_url)
query = Query(fhir_api_url,fhir_rules=fhir_rules)
query.from_config(config)<jupyter_output><empty_output><jupyter_text>3. Request the execution of the query<jupyter_code>query.execute()<jupyter_output><empty_output><jupyter_text>What does this function do?  3.1 Storing info from the config file in a graph Below we can see a visualization of the relationships that have been established between the aliases.<jupyter_code>query.graph_query.draw_relations()
edge = list(query.graph_query.resources_alias_graph.edges)[0]
edge_info = query.graph_query.resources_alias_graph.edges[edge]["info"]

md(
    f"Each edge contains a lot of information. We will illustrate this by the content of the edge between **{edge[0]}** and **{edge[1]}**.<br>"
    f" - the alias of the **parent** resource: *{edge_info.parent}*<br>"
    f" - the alias of the **child** resource: *{edge_info.child}*<br>"
    f" - the **expression** in an instance of *{edge_info.parent}* resource-type which must indicate a reference identical to the id of *{edge_info.child}* resource-type: *{edge_info.searchparam_parent}*<br>"
    f" - the **prefix** that must be added **in front of a param search** in the url of a query sent to a FHIR API if the request is made from an *{edge_info.parent}* resource-type but the search parameter relates to a *{edge_info.child}* resource-type : "
    f"*{edge_info.searchparam_prefix[edge_info.parent]}*<br>"
    f" - also the reverse **prefix** from a *{edge_info.child}* resource-type to a *{edge_info.parent}* resource-type : "
    f"*{edge_info.searchparam_prefix[edge_info.child]}*<br>"
    "the other information is not useful for the moment"
)<jupyter_output><empty_output><jupyter_text>Each node also contains several pieces of information. This information can be found in the dictionary: *query.graph_query.resources_alias_info*<jupyter_code>maxi_len = 0
maxi_alias = None
maxi_elements = None
resource_type= None
for resource_alias, node_info in query.graph_query.resources_alias_info.items():
    curr_len = len(node_info.elements.elements)
    if curr_len > maxi_len:
        maxi_len = curr_len
        maxi_alias = resource_alias
        maxi_elements = node_info.elements
        resource_type = node_info.resource_type

elements_str = custom_repr(maxi_elements.__repr__(), breakline_char="<br>", indent_char="·").replace("[","\[").replace("]","\]")
md(
    f"For example, node *{maxi_alias}* contains the following information: <br>"
    f" - the **type of the associated resource**: *{resource_type}*<br>"
    f" - the **the elements** to be retrieved in the json format of the instances corresponding to the query: {elements_str}<br>"
    f" - the **forest of process trees** which shows the sequence of operations that must be performed on a resource of type *{resource_type}* to retrieve the elements listed above. The position of an item in the list above allows you to find it in the forest below."
)
for tree in maxi_elements.forest.trees.values():
    show_tree(tree.graph)<jupyter_output><empty_output><jupyter_text>3.2 Making calls to the api for each node of the graph<jupyter_code>print(f"The url queries that are made for each nodes are:")
for resource_alias in query.graph_query.resources_alias_info.keys():
    url_builder = URLBuilder(
                    fhir_api_url=fhir_api_url,
                    graph_query=query.graph_query,
                    main_resource_alias=resource_alias,
                )
    print(f" - {resource_alias} → {url_builder.compute()}")<jupyter_output>The url queries that are made for each nodes are:
 - allergy → http://hapi.fhir.org/baseR4/AllergyIntolerance?patient%3APatient.birthdate=1976&asserter%3APractitioner.address-city=paris&code=http%3A%2F%2Fsnomed.info%2Fsct%7C22749300
 - patient → http://hapi.fhir.org/baseR4/Patient?_has%3ACondition%3Asubject%3Acode=http%3A%2F%2Fsnomed.info%2Fsct%7C386661006&_has%3AAllergyIntolerance%3Apatient%3Acode=http%3A%2F%2Fsnomed.info%2Fsct%7C22749300&birthdate=1976
 - practitioner → http://hapi.fhir.org/baseR4/Practitioner?_has%3AAllergyIntolerance%3Aasserter%3Acode=http%3A%2F%2Fsnomed.info%2Fsct%7C22749300&address-city=paris
 - condition → http://hapi.fhir.org/baseR4/Condition?subject%3APatient.birthdate=1976&code=http%3A%2F%2Fsnomed.info%2Fsct%7C386661006<jupyter_text>3.3 Storing query results in multiple dataframes (1 dataframe per alias)<jupyter_code>for _,df in query.dataframes.items():
    display(df.head(5))<jupyter_output><empty_output><jupyter_text>3.4 Joining the dataframes by following an eulerian path in the graph The dataframes of each alias will be linked in the following order.This order corresponds to a Eulerian path in the query graph and thus guarantees that joins between the main dataframe (left alias) and the new dataframe (right alias) will be possible.<jupyter_code>pp.pprint(join_path(query.graph_query.resources_alias_graph))<jupyter_output>[('allergy', 'practitioner'), ('allergy', 'patient'), ('patient', 'condition')]<jupyter_text>Display the result of the query<jupyter_code>df = query.main_dataframe
df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11064543168482817, 'fraction_numerical': 0.011525565800502935, 'mean_word_length': 4.763687600644122, 'pattern_counts': {'":': 0, '<': 94, '<?xml version=': 0, '>': 95, 'https://': 6, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '631650', 'license': 'Apache-2.0', 'n_tokens_mistral': 4583, 'n_tokens_neox': 4406, 'n_words': 1285, 'path': 'examples/tutorials_generator.ipynb', 'repo_name': 'arkhn/FHIR2Dataset', 'size': '14316'}