<jupyter_start><jupyter_text>NOTE: THIS EXPLORATION IS INCOMPLETE - WE ONLY HIGHLIGHT A FEW ASPECTS THAT SHOULD NOT BE MISSED  Audit Data / ExplorationObjectives:- Screen variables and data set- Understand relationship between variables- Understand input-output-relationships- Identify necessary processing steps to build a modelNotes:- Data source and background information https://archive.ics.uci.edu/ml/datasets/Audit+Data.- Above link also references a **paper from the authors/owners of the data set**, please make sure you read it.- There are two data sets, "audit_risk" and "trial". The former is more comprehensive, both can be used for the projects.- The data set features some interesting properties often seen in semi-automated- or white-box-settings.*The discussion herein focuses on the "trial" data set but is applicable to "audit_risk" as well.*<jupyter_code>import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
from sklearn.model_selection import train_test_split

%matplotlib inline
audit_risk = pd.read_csv('https://raw.githubusercontent.com/caichinger/MLMNC2020/master/projects/AuditDataDataSet/data/audit_risk.csv')
trial = pd.read_csv('https://raw.githubusercontent.com/caichinger/MLMNC2020/master/projects/AuditDataDataSet/data/trial.csv')<jupyter_output><empty_output><jupyter_text>Preprocessing<jupyter_code>df = trial.copy()
df.dtypes
df.LOCATION_ID.value_counts()
location_id, _ = pd.factorize(df['LOCATION_ID'])
df['LOCATION_ID'] = location_id
df.isnull().sum()
df = df.dropna()<jupyter_output><empty_output><jupyter_text>Variable InspectionWhen inspecting the scatterplots, one can easily verify that some variables are directly depending on others. One example is given below.<jupyter_code>df.plot.scatter(y='SCORE_A', x='PARA_A')<jupyter_output><empty_output><jupyter_text>These relationships also become apparent in the correlation matrix.<jupyter_code>df.corr('spearman').style.background_gradient().format('{:.2f}')<jupyter_output><empty_output><jupyter_text>Yet, to untangle causation and correlation, a different can approach help. We let a model explain how well one variable explains another.Clearly, considering the variable groups already discovered through the correlation matrix and inspecting them through plots serves the same purpose.In below example, we use a random forest. Knowing that it does not easily overfit, we skip the train-test-split for convenience. Note that we still focus on data exploration and employ a little bit of modelling to help with that.<jupyter_code>variables = df.columns  # df works as well

x_explains_y = pd.DataFrame(index=variables, columns=variables, dtype=np.float)
for xname in variables:
    for yname in variables:
        X = df[[xname]].values
        y = df[yname]
        
        rf = RandomForestRegressor(n_estimators=50, oob_score=True)
        rf.fit(X, y)
        
        x_explains_y.loc[xname, yname] = rf.oob_score_  # out of bag store ~ test score
        
x_explains_y.head(2)
x_explains_y.style.background_gradient(low=0, high=1).format('{:.2f}')
# for each row, read this table left to right
# the number indicates how well the row-variable explains the column-variable
# this is similar to a correlation matrix only that the relationship between 
# the two variables can be more complex<jupyter_output><empty_output><jupyter_text>We again observe that SCORE_A is only an aggregate of PARA_A and this behvior holds true in other cases as well.Conclusions:- We observe that a number of variable depend on each other and hence only a subset of variables is truly  independent.- We select the subset of truly independent variables for modelling purposes.- We also need to take care to not accidentially include a target varible as explanatory variable. In addition, focusing on PARA, SCORE and TOTAL, we see that TOTAL is merely a sum and hence carries little meaning by itself.<jupyter_code>df[['PARA_A', 'SCORE_A', 'PARA_B', 'SCORE_B', 'TOTAL']].head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07957041737856968, 'fraction_numerical': 0.00366121552355382, 'mean_word_length': 5.403125, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '201657', 'license': 'MIT', 'n_tokens_mistral': 1223, 'n_tokens_neox': 1178, 'n_words': 431, 'path': 'projects/AuditDataDataSet/data_exploration.ipynb', 'repo_name': 'caichinger/MLMNC2020', 'size': '4097'}