<jupyter_start><jupyter_text>Get datasets for using only the site data and using all (site, filed, outside) data subscript 1 means only using the site data subscript 2 means using all (site, filed, outside) data Prerequest: need to mannuly change "only_site_data" in "dataset.pu::get_judgement_calls_dictionary()" to collect the datasets.<jupyter_code>feats1 = ['AlteredMentalStatus', 'FocalNeuroFindings', 'Torticollis', 'PainNeck', 'TenderNeck', 'PosMidNeckTenderness', 'PtCompPainHead', 'PtCompPainFace', 'PtCompPainExt', 'PtCompPainTorsoTrunk', 'PtTenderHead', 'PtTenderFace', 'PtTenderExt', 'PtTenderTorsoTrunk', 'SubInj_Head', 'SubInj_Face', 'SubInj_Ext', 'SubInj_TorsoTrunk', 'Immobilization', 'MedsRecd', 'ArrPtIntub']
feats2 = ['AlteredMentalStatus2', 'FocalNeuroFindings2', 'Torticollis2', 'PainNeck2', 'TenderNeck2', 'PosMidNeckTenderness2', 'PtCompPainHead2', 'PtCompPainFace2', 'PtCompPainExt2', 'PtCompPainTorsoTrunk2', 'PtTenderHead2', 'PtTenderFace2', 'PtTenderExt2', 'PtTenderTorsoTrunk2', 'subinj_Head2', 'subinj_Face2', 'subinj_Ext2', 'subinj_TorsoTrunk2', 'Immobilization2', 'MedsRecd2', 'ArrPtIntub2']

raw_data_path_1_train = oj(rulevetting.DATA_PATH, Dataset().get_dataset_id(), 'judgement_call/only_site_True/train.csv')
raw_data_path_2_train  = oj(rulevetting.DATA_PATH, Dataset().get_dataset_id(), 'judgement_call/only_site_False/train.csv')

raw_data_path_1_tune  = oj(rulevetting.DATA_PATH, Dataset().get_dataset_id(), 'judgement_call/only_site_True/tune.csv')
raw_data_path_2_tune = oj(rulevetting.DATA_PATH, Dataset().get_dataset_id(), 'judgement_call/only_site_False/tune.csv')

raw_data_path_1_test = oj(rulevetting.DATA_PATH, Dataset().get_dataset_id(), 'judgement_call/only_site_True/test.csv')
raw_data_path_2_test= oj(rulevetting.DATA_PATH, Dataset().get_dataset_id(), 'judgement_call/only_site_False/test.csv')

df_1_train, df_1_tune, df_1_test = pd.read_csv(raw_data_path_1_train),pd.read_csv(raw_data_path_1_tune),pd.read_csv(raw_data_path_1_test)
df_1=pd.concat([df_1_train, df_1_tune, df_1_test])
df_2_train, df_2_tune, df_2_test = pd.read_csv(raw_data_path_2_train),pd.read_csv(raw_data_path_2_tune),pd.read_csv(raw_data_path_2_test)
df_2=pd.concat([df_2_train, df_2_tune, df_2_test])
df_1=df_1[feats1]
df_2=df_2[feats2]<jupyter_output><empty_output><jupyter_text>The difference between using only the site data and using all (site, filed, outside) data The returned number is the percentage of the difference for each features<jupyter_code>df_1_sum=df_1.sum(axis=0)/df_1.shape[0]*100
df_2_sum=df_2.sum(axis=0)/df_1.shape[0]*100
df_2_sum.index = feats1
df_difference=df_2_sum.subtract(df_1_sum)
print(df_difference)<jupyter_output>AlteredMentalStatus     6.881980
FocalNeuroFindings      3.682463
Torticollis             0.965892
PainNeck                4.799276
TenderNeck              2.776939
PosMidNeckTenderness    2.143073
PtCompPainHead          7.546031
PtCompPainFace          2.263809
PtCompPainExt           4.255961
PtCompPainTorsoTrunk    5.463326
PtTenderHead            2.565650
PtTenderFace            0.754603
PtTenderExt             1.811047
PtTenderTorsoTrunk      3.380622
SubInj_Head             3.501358
SubInj_Face             0.724419
SubInj_Ext              1.660127
SubInj_TorsoTrunk       2.263809
Immobilization          0.814971
MedsRecd                0.271657
ArrPtIntub              0.000000
dtype: float64
{'fraction_non_alphanumeric': 0.10798816568047337, 'fraction_numerical': 0.06686390532544378, 'mean_word_length': 5.92827868852459, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '630740', 'license': 'MIT', 'n_tokens_mistral': 1448, 'n_tokens_neox': 1325, 'n_words': 201, 'path': 'rulevetting/projects/csi_pecarn/notebooks/different_on_site_or_all_data.ipynb', 'repo_name': 'csinva/medical-rules', 'size': '3380'}