<jupyter_start><jupyter_text>Adversarial Validation**Description**The objective of `Adversarial Validation` is to learn a model that predicts which rows are in the training dataset, and which are in the test set. Thi classifier model will try to predict which data rows are from the training set, and which are from the test set.If the two datasets came from the same distribution, this should be impossible. In such situation, if we attempted to train a classifier to distinguish training examples from test examples, it would perform no better than random. This would correspond to ROC AUC of 0.5. But if there are systematic differences in the feature values of your training and test datasets, then a classifier will be able to successfully learn to distinguish between them.The better a model you can learn to distinguish them, the bigger the problem you have.**Pre-requisites**- Modules: See [requirements.txt](./requirements.txt) for specific versions I used**Notes**- To get the Catboost training GUI, you might need to [install/enable ipywidgets](https://catboost.ai/docs/installation/python-installation-additional-data-visualization-packages.html) Sources<jupyter_code># Expand the paragraph frames
from IPython.core.display import display, HTML
display(HTML("<style>.container { width:90% !important; }</style>"))<jupyter_output><empty_output><jupyter_text>1. Data Preparation**Loading parameters & function**<jupyter_code>import os
import pandas as pd

# Loads the dataset
data_dir = './data' # location of unzipped CSVs
df_train = pd.read_csv( os.path.join(data_dir, 'train.csv') )
df_test = pd.read_csv( os.path.join(data_dir, 'test.csv') )<jupyter_output><empty_output><jupyter_text>**Only keeping numerical columns**<jupyter_code>numeric_cols = [
    'payRecMkr', 'payMethod', 'accHolderType', 'amount', 'txMonth', 'txDay', 'txWeekDay', 'logAmount', 'txTypeCode', 'payRecMkrCode', 'payMethodCode', 'accHolderTypeCode'
]

df_train = df_train[numeric_cols]
df_test = df_test[numeric_cols]<jupyter_output><empty_output><jupyter_text>**Filling missing values**<jupyter_code># Numeric with -999
df_train = df_train.fillna(-999)
df_test = df_test.fillna(-999)

df_train.drop_duplicates(inplace=True)
df_test.drop_duplicates(inplace=True)
df_train['dataset_label'] = 0
df_train['dataset_label'] = 1<jupyter_output><empty_output><jupyter_text>**Visualise dataset contents**<jupyter_code>df_train.head()
df_test.head()<jupyter_output><empty_output><jupyter_text>**Create adversarial label**Indicates whether sample is from test set or not<jupyter_code>df_train['is_test'] = 0
df_test['is_test'] = 1

orig_df_train = df_train.copy()

frames = [orig_df_train, df_test]
df_train = pd.concat(frames)

df_train['target'] = df_train.is_test
df_train.drop( 'is_test', axis = 1, inplace = True )<jupyter_output><empty_output><jupyter_text>**Cross-validating logistic regression**<jupyter_code>import warnings

from sklearn.model_selection import cross_validate as CV
from sklearn.linear_model import LogisticRegression as LR

warnings.filterwarnings("ignore")

clf = LR()

scores = CV( clf, df_train.drop( 'target', axis = 1 ), df_train.target, scoring = ('accuracy', 'roc_auc'), cv = 5, verbose = 1, return_estimator = True )
sorted(scores.keys())
#print("mean AUC: {:.2%}, std: {:.2%} \n".format( scores.mean(), scores.std()))
print( scores['test_score'] )
print( scores )
from sklearn.ensemble import RandomForestClassifier as RF
from sklearn.metrics import roc_auc_score as AUC
from sklearn.metrics import accuracy_score as accuracy<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11160964666292765, 'fraction_numerical': 0.006730229949523275, 'mean_word_length': 6.8915929203539825, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '624308', 'license': 'MIT', 'n_tokens_mistral': 1165, 'n_tokens_neox': 1114, 'n_words': 330, 'path': 'AdversarialValidation/adversarial-validation.ipynb', 'repo_name': 'mdehaut/explainability', 'size': '3566'}