<jupyter_start><jupyter_text>Example 05: Causal discovery with latent unobserved variables------------------------------------------- Overview - AitiaExplorer allows you to see if there are latent, unoberved variables in the discovered causal graphs. - In the example below, we will see how AitiaExplorer shows these latent variables.  Imports<jupyter_code>import os
import sys
import math
import networkx as nx
import matplotlib.pyplot as plt
from pycausal.pycausal import pycausal
from IPython.display import display, HTML

module_path = os.path.abspath(os.path.join('../src'))
if module_path not in sys.path:
    sys.path.append(module_path)
from aitia_explorer.app import App

# stop the warning clutter
import warnings
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>Set up for causal discovery- First create an AitiaExplorer instance and using it to load some data into a pandas dataframe.- We will be using the [HEPAR II](https://www.bnlearn.com/bnrepository/hepar2) dataset.<jupyter_code>aitia = App()
df = aitia.data.hepar2_10k_data()
df.head()<jupyter_output><empty_output><jupyter_text>Set up for causal discovery- Next we will select the causal and feature selection algorithms and start the py-causal Java VM.<jupyter_code>feature_selection_list = []
feature_selection_list.append(aitia.feature_selection.LINEAR_REGRESSION)
feature_selection_list.append(aitia.feature_selection.RANDOM_FOREST)
algorithm_list = []
algorithm_list.append(aitia.algo_runner.PC)
algorithm_list.append(aitia.algo_runner.RFCI_continuous)
pc = pycausal()
pc.start_vm()<jupyter_output><empty_output><jupyter_text>- Now we will run the causal discovery with the number of feature set to a 30.<jupyter_code>results, summary, target_graph = aitia.run_analysis( 
                                       df,
                                       target_graph_str=None, # no target graph, force system to create one
                                       n_features=30,
                                       feature_selection_list=feature_selection_list,
                                       algorithm_list=algorithm_list,
                                       pc=pc)<jupyter_output>No target graph has been supplied.
The system will generate an approximate target graph using the greedy hill climbing algorithm.
Running causal discovery on features selected by LINEAR_REGRESSION
There are 2 latent edges in the reduced dataset
Running causal discovery using PC
Running causal discovery using RFCI-continuous
Completed causal discovery on features selected by LINEAR_REGRESSION
Running causal discovery on features selected by RANDOM_FOREST
There are 0 latent edges in the reduced dataset
Running causal discovery using PC
Running causal discovery using RFCI-continuous
Completed causal discovery on features selected by RANDOM_FOREST
Completed analysis.<jupyter_text>Display the results- As we can see above the output from the analysis tells us there are latent variables.- We can display the summary dataframe to get more details on the results.<jupyter_code>display(HTML(summary.to_html()))<jupyter_output><empty_output><jupyter_text>- It looks like the results from the RFCI / linear algebra run produced some latent variables.- We can view the causal graph with these latent, unobserved variables using the results object.<jupyter_code>results[1].causal_graph_with_latent_edges.draw()<jupyter_output><empty_output><jupyter_text>- We can also get a list of the edges where these missing nodes are.<jupyter_code>results[1].latent_edges<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07005302818866871, 'fraction_numerical': 0.003907340217694669, 'mean_word_length': 4.055007052186178, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '396581', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 1017, 'n_tokens_neox': 952, 'n_words': 339, 'path': 'notebooks/Example05-Causal_Discovery_With_Latent _Variables.ipynb', 'repo_name': 'corvideon/aitiaexplorer', 'size': '3583'}