<jupyter_start><jupyter_text>Example 04: Causal discovery within a range of features------------------------------------------- Overview - AitiaExplorer allows you to set a higher and lower number of features for the causal discovery process. - In the example below, we set a number range and AitiaExplorer will start at the lower number of features and run the causal discovery analysis for each number in the range. The best performing results will then be returned. - This example is designed to show how AitiaExplorer can help narrow the choices for causal discovery.  Imports<jupyter_code>import os
import sys
import math
import networkx as nx
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from pycausal.pycausal import pycausal
from IPython.display import display, HTML

module_path = os.path.abspath(os.path.join('../src'))
if module_path not in sys.path:
    sys.path.append(module_path)
from aitia_explorer.app import App

# stop the warning clutter
import warnings
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>Set up for causal discovery- First create an AitiaExplorer instance and using it to load some data into a pandas dataframe.- We will be using the [HEPAR II](https://www.bnlearn.com/bnrepository/hepar2) dataset.<jupyter_code>aitia = App()
df = aitia.data.hepar2_10k_data()
df.head()<jupyter_output><empty_output><jupyter_text>Set up for causal discovery- Next we will select the causal and feature selection algorithms and start the py-causal Java VM.<jupyter_code>feature_selection_list = []
feature_selection_list.append(aitia.feature_selection.LINEAR_REGRESSION)
feature_selection_list.append(aitia.feature_selection.RANDOM_FOREST)
algorithm_list = []
algorithm_list.append(aitia.algo_runner.PC)
algorithm_list.append(aitia.algo_runner.FGES_continuous)
pc = pycausal()
pc.start_vm()<jupyter_output><empty_output><jupyter_text>- Now we will run the causal discovery with the number of feature set to a range between 10 and 20.<jupyter_code>results, summary, target_graph, all_results = aitia.run_analysis_with_high_low( 
                                               df,
                                               target_graph_str=None, # no target graph, one will be created
                                               feature_high=20, feature_low=10,
                                               feature_selection_list=feature_selection_list,
                                               algorithm_list=algorithm_list,
                                               pc=pc)<jupyter_output>No target graph has been supplied.
The system will generate an approximate target graph using the greedy hill climbing algorithm.
-----------------------------------------------
Starting analysis with 20 features...
Running causal discovery on features selected by LINEAR_REGRESSION
There are 0 latent edges in the reduced dataset
Running causal discovery using PC
Running causal discovery using FGES-continuous
Completed causal discovery on features selected by LINEAR_REGRESSION
Running causal discovery on features selected by RANDOM_FOREST
There are 0 latent edges in the reduced dataset
Running causal discovery using PC
Running causal discovery using FGES-continuous
Completed causal discovery on features selected by RANDOM_FOREST
Completed analysis.
Completed analysis with 20 features...
-----------------------------------------------
Starting analysis with 19 features...
Running causal discovery on features selected by LINEAR_REGRESSION
There are 0 latent edges in the reduced dataset
Ru[...]<jupyter_text>Display the results- The summary dataframe will contain the analysis results with the lowest SHD.<jupyter_code>display(HTML(summary.to_html()))<jupyter_output><empty_output><jupyter_text>- We can look at the causal graph for the best result (index 2 or 3).<jupyter_code>results[2].causal_graph.draw()<jupyter_output><empty_output><jupyter_text>- We can take a look at all the results.<jupyter_code>display(HTML(all_results.to_html()))<jupyter_output><empty_output><jupyter_text>- We can now plot the SHD and AUPRC for the run.<jupyter_code>report_df = all_results.copy()
report_df.columns = [column.replace(" ", "_") for column in all_results.columns] 
plt.rcParams["figure.figsize"] = (10, 5)

plt.plot(np.arange(11,21), 
         report_df.query('Causal_Algorithm == "PC" and Feature_Selection_Method == "LINEAR_REGRESSION"')['SHD'], 
         label='PC / Linear Regression', color='red')
plt.plot(np.arange(11,21),  
         report_df.query('Causal_Algorithm == "PC" and Feature_Selection_Method == "RANDOM_FOREST"')['SHD'], 
         label='PC / Random Forest', color='green')
plt.plot(np.arange(11,21), 
         report_df.query('Causal_Algorithm == "FGES-continuous" and Feature_Selection_Method == "LINEAR_REGRESSION"')['SHD'], 
         label='FGES / Linear Regression', color='blue')
plt.plot(np.arange(11,21),  
         report_df.query('Causal_Algorithm == "FGES-continuous" and Feature_Selection_Method == "RANDOM_FOREST"')['SHD'], 
         label='FGES / Random Forest', color='black')

plt.title('PC vs FGES - Structural Hamming Distance')
plt.xlabel('Number of Features')
plt.ylabel('SHD')
plt.legend()
plt.show()

plt.plot(np.arange(11,21), 
         report_df.query('Causal_Algorithm == "PC" and Feature_Selection_Method == "LINEAR_REGRESSION"')['AUPRC'], 
         label='PC / Linear Regression', color='red')
plt.plot(np.arange(11,21),  
         report_df.query('Causal_Algorithm == "PC" and Feature_Selection_Method == "RANDOM_FOREST"')['AUPRC'], 
         label='PC / Random Forest', color='green')
plt.plot(np.arange(11,21), 
         report_df.query('Causal_Algorithm == "FGES-continuous" and Feature_Selection_Method == "LINEAR_REGRESSION"')['AUPRC'], 
         label='FGES / Linear Regression', color='blue')
plt.plot(np.arange(11,21),  
         report_df.query('Causal_Algorithm == "FGES-continuous" and Feature_Selection_Method == "RANDOM_FOREST"')['AUPRC'], 
         label='FGES / Random Forest', color='black')
plt.title('PC vs FGES  - AUPRC')
plt.xlabel('Number of Features')
plt.ylabel('AUPRC')
plt.legend()
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11662892267874474, 'fraction_numerical': 0.00986735684244581, 'mean_word_length': 3.7745173745173743, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '141181', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 1892, 'n_tokens_neox': 1746, 'n_words': 515, 'path': 'notebooks/Example04-Causal_Discovery_Within_A_Range_of_Features.ipynb', 'repo_name': 'corvideon/aitiaexplorer', 'size': '6182'}