<jupyter_start><jupyter_text>Given that from usaual mathimatics and the corrilation we did in EDA part we see  radius which can affect thevalue of perimeter,area, compactness,concavity,smoothness and so on.So using this we can formulate questions:        1. if radius_mean getting low then breast tumors being diagnosed as benign.        2. if texture dencity increase then breast tumors being diagnosed.<jupyter_code>df['radius_min'] = df['radius_mean'].apply(lambda x: True if x > 19 else False)

train,test = train_test_split(df,test_size=0.1)
train_features = {'diagnosis','radius','area','perimeter','compactness','smoothness','concave points','concavity'}
train_columns = [col for col in df 
                      if any(feature in col for feature in train_features)]
train_columns<jupyter_output><empty_output><jupyter_text>The stronger the correlation, the more difficult it is to change one variable without changing another. It becomes difficult for the model to estimate the relationship between each independent variable and the dependentvariable independentlybecause the independent variables tend to change in unison. So we need to drop features that have high positive corrilation.<jupyter_code>df.corr()
training = df[train_columns]
training = training.drop(training.filter(regex = '_worst').columns,axis=1)
training = training.drop(training.filter(regex = '_se').columns,axis=1)
training.rename(columns={'concave points_mean':'concave_points_mean'},inplace=True)
training.head()
causal_graph = """
digraph{
radius_mean;
perimeter_mean;
area_mean;
smoothness_mean;
compactness_mean;
concavity_mean;
concave_points_mean;
U[label="Unobserved Confounders"];
radius_mean -> perimeter_mean;radius_mean -> area_mean;radius_mean -> smoothness_mean;radius_mean->radius_min;area_mean->compactness_mean;
perimeter_mean->compactness_mean;concave_points_mean->concavity_mean;
U->compactness_mean;U->smoothness_mean;U->concavity_mean;U->radius_min;U->diagnosis;
compactness_mean->diagnosis;smoothness_mean->diagnosis;concavity_mean->diagnosis;radius_min->diagnosis;
U->concave_points_mean;radius_mean ->U
}
"""
model= CausalModel(
        data = training,
        graph=causal_graph.replace("\n", " "),
        treatment='radius_min',
        outcome='diagnosis')
model.view_model()
display(Image(filename="causal_model.png"))

#causal effect
estimands = model.identify_effect()
print(estimands)
#Causal Effect Estimation using statistical method
estimate = model.estimate_effect(estimands,method_name = "backdoor.propensity_score_weighting")
print(estimate)
#Logistic regression estimate
import statsmodels.api as sm
causal_estimate = model.estimate_effect(estimands,
                                        method_name="backdoor.generalized_linear_model",
                                       confidence_intervals=False,
                                       test_significance=False,
                                        method_params = {
                                            'num_null_simulations':10,
                                            'num_simulations':10,
                                            'num_quantiles_to_discretize_cont_cols':10,
                                            'fit_method': "statsmodels",
                                            'glm_family': sm.families.Binomial(), # logistic regression
                                            'need_conditional_estimates':False
                                        },
                                       )
print(causal_estimate)
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LassoCV
from sklearn.ensemble import GradientBoostingRegressor
dml_estimate = model.estimate_effect(identified_estimand, method_name="backdoor.econml.dml.DML",
                control_value = 0,
                treatment_value = 1,
                target_units = lambda df: df["X0"]>1,
                confidence_intervals=False,
                method_params={
                    "init_params":{'model_y':GradientBoostingRegressor(),
                                   'model_t': GradientBoostingRegressor(),
                                   'model_final':LassoCV(),
                                   'featurizer':PolynomialFeatures(degree=1, include_bias=True)},
                    "fit_params":{}}
                                        )<jupyter_output><empty_output><jupyter_text>Testing  validity of our assumption<jupyter_code>#Random Common Cause
#adds an independent random variable as a common cause to the train data
#if the assumption was correct, the estimation should not change.

refute_train = model.refute_estimate(estimands,estimate, "random_common_cause")
print(refute_train)
#Data Subset Refuter
#replaces the given dataset with a randomly selected subset
#if the assumption was correct, the estimation should not change that much.

refute_train = model.refute_estimate(estimands,estimate,"data_subset_refuter")
print(refute_train)
0.6612617333976643 - 0.6604384423272592
#Placebo Treatment
#replaces the true treatment variable with an independent random variable
#if the assumption was correct, the estimate should go close to zero.

refute_train = model.refute_estimate(estimands,estimate, "placebo_treatment_refuter")
print(refute_train)<jupyter_output>Refute: Use a Placebo Treatment
Estimated effect:-0.6604384423272592
New effect:0.0010624673849260824
p value:0.47<jupyter_text>Conclusion Observing the three treatment above we can say that for the "mean" features, low radius mean has a causal effect on the breast cancer diagnosis  2. Standard error casual effect A high standard error shows that sample means are widely spread around the population mean—your samplemay not closely represent your population'Assumption: if texture standard error .cancer being benign.<jupyter_code>df['texture_se'].describe()
train,test = train_test_split(df,test_size=0.1)
train_features = {'diagnosis','radius','area','perimeter','compactness','smoothness','concave points','concavity','texture'}
train_columns = [col for col in df 
                      if any(feature in col for feature in train_features)]
train_columns
df['texture_min'] = df['texture_se'].apply(lambda x: True if x < 1.5 else False)
training = df[train_columns]
training = training.drop(training.filter(regex = '_worst').columns,axis=1)
training = training.drop(training.filter(regex = '_mean').columns,axis=1)
training.rename(columns={'concave points_se':'concave_points_se'},inplace=True)
training.head()
causal_graph = """
digraph{
radius_se;
perimeter_se;
area_se;
smoothness_se;
compactness_se;
concavity_se;
concave_points_se;
U[label="Unobserved Confounders"];
radius_se -> perimeter_se;radius_se -> area_se;radius_se -> smoothness_se;radius_se->texture_min;area_se->compactness_se;
perimeter_se->compactness_se;concave_points_se->concavity_se;
U->compactness_se;U->smoothness_se;U->concavity_se;U->texture_min;U->diagnosis;
compactness_se->diagnosis;smoothness_se->diagnosis;concavity_se->diagnosis;texture_min->diagnosis;
U->concave_points_se;radius_se ->U
}
"""
model_1= CausalModel(
        data = training,
        graph=causal_graph.replace("\n", " "),
        treatment='texture_min',
        outcome='diagnosis')
model_1.view_model()
display(Image(filename="causal_model.png"))
#causal effect
estimands = model_1.identify_effect()
print(estimands)
#Causal Effect Estimation using statistical method
estimate = model_1.estimate_effect(estimands,method_name = "backdoor.propensity_score_weighting")
print(estimate)
#Logistic regression estimate
import statsmodels.api as sm
causal_estimate = model_1.estimate_effect(estimands,
                                        method_name="backdoor.generalized_linear_model",
                                       confidence_intervals=False,
                                       test_significance=False,
                                        method_params = {
                                            'num_null_simulations':10,
                                            'num_simulations':10,
                                            'num_quantiles_to_discretize_cont_cols':10,
                                            'fit_method': "statsmodels",
                                            'glm_family': sm.families.Binomial(), # logistic regression
                                            'need_conditional_estimates':False
                                        },
                                       )
print(causal_estimate)<jupyter_output>*** Causal Estimate ***

## Identified estimand
Estimand type: nonparametric-ate

### Estimand : 1
Estimand name: backdoor
Estimand expression:
      d                                                                       
─────────────(Expectation(diagnosis|area_se,compactness_se,concave_points_se,r
d[textureₘᵢₙ]                                                                 

                                                  
adius_se,perimeter_se,smoothness_se,concavity_se))
                                                  
Estimand assumption 1, Unconfoundedness: If U→{texture_min} and U→diagnosis then P(diagnosis|texture_min,area_se,compactness_se,concave_points_se,radius_se,perimeter_se,smoothness_se,concavity_se,U) = P(diagnosis|texture_min,area_se,compactness_se,concave_points_se,radius_se,perimeter_se,smoothness_se,concavity_se)

## Realized estimand
b: diagnosis~Sigmoid(texture_min+area_se+compactness_se+concave_points_se+radius_se+perimeter_se+smoothness_se+concavity_se+te[...]<jupyter_text>Testing validity of our assumption<jupyter_code>#Random Common Cause
#adds an independent random variable as a common cause to the train data
#if the assumption was correct, the estimation should not change.

refute_train = model_1.refute_estimate(estimands,estimate, "random_common_cause")
print(refute_train)
#Data Subset Refuter
#replaces the given dataset with a randomly selected subset
#if the assumption was correct, the estimation should not change that much.

refute_train = model_1.refute_estimate(estimands,estimate,"data_subset_refuter")
print(refute_train)
#Placebo Treatment
#replaces the true treatment variable with an independent random variable
#if the assumption was correct, the estimate should go close to zero.

refute_train = model_1.refute_estimate(estimands,estimate, "placebo_treatment_refuter")
print(refute_train)<jupyter_output>Refute: Use a Placebo Treatment
Estimated effect:-0.07407164116266085
New effect:0.0003115969614423153
p value:0.47
{'fraction_non_alphanumeric': 0.08482948608340458, 'fraction_numerical': 0.01491403058801178, 'mean_word_length': 3.217948717948718, 'pattern_counts': {'":': 2, '<': 19, '<?xml version=': 0, '>': 56, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 49, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '536316', 'license': 'MIT', 'n_tokens_mistral': 3069, 'n_tokens_neox': 2851, 'n_words': 643, 'path': 'notebooks/Casual_inference.ipynb', 'repo_name': 'smegnshd/Breast-Cancer', 'size': '10527'}