<jupyter_start><jupyter_text>Plot calibration curve of original model<jupyter_code># load results from the aucroc evaluated model)
pred_df = pd.read_csv('./roc_results/2021_xgb_pooling_results_final_roc.csv')
pred_df = pred_df.loc[:,['averaged','died_in_90','usrds_id']]
pred_df.head()
onc_plot_calibration_curve(
                y_true=pred_df.died_in_90, 
                y_proba=pred_df.averaged, 
                label='XGBoost_imputed',
                filename='./roc_results/xgb_imputed_orig_calibration')<jupyter_output><empty_output><jupyter_text>Calibrate the model The XGBoost model can be calibrated by training an isotonic regression on a portion of the testing set.<jupyter_code># get subset of each row for splitting
df = pd.read_sql_query('''SELECT   usrds_id, subset FROM medxpreesrd;''', con)
df.head()
data = pd.merge(pred_df, df, how="left", on="usrds_id")
data = data.rename(columns={'died_in_90':'y','averaged':'score'})
data.head()
calibrated_results = calibrate_onc(data, path='./roc_results/',model_name='xgb_imputed')<jupyter_output>accuracy - original/calibrated: 0.9031552911873056 / 0.9268700521672487
ROC AUC - original/calibrated:      0.8270572148370063 / 0.8270580630956539
avg precision - original/calibrated: 0.31194486526045584 / 0.3046086815075088
	Brier: 0.059
{'fraction_non_alphanumeric': 0.11633281972265024, 'fraction_numerical': 0.0901386748844376, 'mean_word_length': 5.27536231884058, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '69426', 'license': 'MIT', 'n_tokens_mistral': 559, 'n_tokens_neox': 468, 'n_words': 84, 'path': 'xgboost/imputed/4_xgb_imputed_calibration.ipynb', 'repo_name': 'onc-healthit/2021PCOR-ML-AL', 'size': '1298'}