<jupyter_start><jupyter_text>Predicting Fraud Medical Claims**Author**: <NAME>The scope of this notebook is to provide instructions on how to run a fraud medical claims use case with Python and DataRobot. The data is ingested through a database so you need to manipulate the code accordingly to fit your own needs but this serves as a prime example of how you can connect DataRobot to your own source systems.  Requirements- Python version 3.7.3-  DataRobot API version 2.19.0. Small adjustments might be needed depending on the Python version and DataRobot API version you are using.Full documentation of the Python package can be found here: https://datarobot-public-api-client.readthedocs-hosted.com  Import LibrariesImporting libraries that will be used in the project.<jupyter_code>import pandas as pd
import numpy as np
import datarobot as dr
import yaml
import time
import json
import requests

from datetime import datetime

pd.options.display.max_columns = 100<jupyter_output><empty_output><jupyter_text>Connecting to DataRobotUsing dr.Client command alongside a yaml configuration file, we can connect to DataRobot. Alternatively, we could pass the token and endpoint variables to achieve the same result.<jupyter_code>dr.Client(config_path="../drconfig.yaml")<jupyter_output><empty_output><jupyter_text>Define some variables for later...Here we are defining some variables that will be used later in the analysisYaml file description and required tags: - token: a DR API key- username: a login to DR- db_address: an IP or a server name- db_name: a database to connect with- db_user: a dadabase login- db_pass: <PASSWORD>- pred_serv_id: a DR prediction server id<jupyter_code>with open("../drconfig.yaml", 'r') as stream:
    creds = yaml.safe_load(stream)
token = creds['token']

project_name = 'Medical_Insurance_Fraud'
target = 'fraud'
metric = 'LogLoss'

base_url = 'YOUR_DATAROBOT_HOSTNAME'<jupyter_output><empty_output><jupyter_text>Overview of Next Steps1. Create data sources    1. Find Microsoft SQL Server jdbc driver among predefined    2. Create a data store    3. Create a data source2. Create and run projects    1. Supervised models    2. Anomaly detection models    3. Supervised with anomaly detection features3. Compare results4. Deployment and scoring  1. Create data sources<jupyter_code>def dr_rest_call(url, req_func, payload=None):
    headers = {'Authorization': f'Token {token}',
               'Content-Type': 'application/json;charset=UTF-8'}
    return req_func(f'{base_url}{url}', headers=headers, json=payload)<jupyter_output><empty_output><jupyter_text>1.A Find Microsoft SQL Server jdbc driver among predefined<jupyter_code>drivers = dr_rest_call('externalDataDrivers', requests.get)

drivers = drivers.json()
drivers_sql_serv = []
for driver in drivers['data']:
    if 'Microsoft SQL Server' in driver['canonicalName']:
        drivers_sql_serv.append([driver['canonicalName'], driver['version'], driver['id']])
drivers_sql_serv
# Use the latest available driver
driver_sql_serv_id = drivers_sql_serv[-1][-1]
driver_sql_serv_id<jupyter_output><empty_output><jupyter_text>1.B Create a data store<jupyter_code>data = {'type': 'jdbc', 
        'canonicalName': 'sql_server_test', 
        'params': {'driverId': driver_sql_serv_id, 
                   'jdbcFields': 
                   [{'name': 'address', 'value': creds['db_address']},
                    {'name': 'databaseName', 'value': creds['db_name']}]
                  }}

data_store_resp = dr_rest_call('externalDataStores', requests.post, payload=data)
# get data store id
data_store = data_store_resp.json()
data_store_id = data_store['id']
data_store_id<jupyter_output><empty_output><jupyter_text>1.C Create a data source<jupyter_code># create a data source based on query 
ad_query = """
select *
from drdemodb1.cfds_demo.Fraud_Medical_Insurance_Fraud
order by npi
"""
data = {'type': 'jdbc', 
        'canonicalName': 'Medical_Insurance_Fraud_query', 
        'params': {'dataStoreId': data_store_id, 
                   'query': ad_query}}

data_src_query_resp = dr_rest_call('externalDataSources', requests.post, payload=data)
data_src_query = data_src_query_resp.json()
data_src_query_id = data_src_query['id']
data_src_query_id<jupyter_output><empty_output><jupyter_text>2. Create and run projectsWe will be creating multiple functions that will be used later on<jupyter_code>def wait_for_proj_id(resp):
    """
    wait for the project creation
    return the project id
    """
    while True:
        resp_stat = requests.get(resp.headers['Location'], 
                                 headers={'Authorization': f'Token {token}', 
                                          'Content-Type': 'application/json;charset=UTF-8'})
        resp_stat = resp_stat.json()

        if resp_stat.get('id') is None:
            time.sleep(10)
        else:
            proj_id = resp_stat.get('id')
            break
        
    return proj_id
def get_model_score(mod, metric):
    res = {}
    res['model_number'] = mod.model_number
    res['model_type'] = mod.model_type
    res['model_category'] = mod.model_category
    res['model'] = mod
    res['sample_pct'] = mod.sample_pct
    
    res['metric_v'] = mod.metrics.get(metric, {}).get('validation')
    res['metric_cv'] = mod.metrics.get(metric, {}).get('crossValidation')
    
    return res

def get_model_scores(proj, metric=None, ascending=True):
    """
    iterate trough the project models and get their performance metric
    """
    if metric is None:
        metric = proj.metric        
    df = pd.DataFrame([get_model_score(m, metric) for m in proj.get_models(with_metric=metric)])
    return df.sort_values(['metric_cv', 'metric_v'], ascending=ascending, na_position='last')
def get_train_preds(mod):
    """
    request and/or retrieve training predictions for a given model
    """
    try:
        # request training predictions and get job ids
        pred_job = mod.request_training_predictions(dr.enums.DATA_SUBSET.ALL)
        preds = pred_job.get_result_when_complete().get_all_as_dataframe()
        return preds
    except:
        # retrieve training predictions if they were already requested
        train_preds = dr.TrainingPredictions.list(mod.project_id)
        for train_pred in train_preds:
            if train_pred.model_id == mod.id and train_pred.data_subset == 'all':
                preds = dr.TrainingPredictions.get(mod.project_id, train_pred.prediction_id).get_all_as_dataframe()
                return preds
def prep_ad_preds(mods):
    """
    preprocess training predictions from anomaly detection models
    mods - a list of aomaly detection models
    """
    preds = get_train_preds(mods[0])
    preds.set_index('row_id', inplace=True)
    preds = preds[['partition_id', 'class_1.0']].copy()
    preds.rename(columns={'class_1.0': f'{mods[0].model_type}_prediction'}, inplace=True)
    for mod in mods[1:]:
        preds_tmp = get_train_preds(mod)
        preds_tmp.set_index('row_id', inplace=True)
        preds = preds.merge(preds_tmp[['class_1.0']], left_index=True, right_index=True)
        preds.rename(columns={'class_1.0': f'{mod.model_type}_prediction'}, inplace=True)
        
    preds['partition_id'] = preds.partition_id.replace('Holdout', '5.0').astype(float).astype(int)
    return preds<jupyter_output><empty_output><jupyter_text>2.A Supervised Models<jupyter_code># create a project based on the data source
data = {'projectName': f'{project_name}_clf',
        'dataSourceId': data_src_query_id, 
        'user': creds['db_user'],
        'password': creds['<PASSWORD>']}

project_clf_resp = dr_rest_call('projects', requests.post, payload=data)

project_clf_id = wait_for_proj_id(project_clf_resp)
# find the project trough python API 
projects = dr.Project.list()
project_clf = [pr for pr in projects if pr.id == project_clf_id][0]
# set target and run autopilot
project_clf.set_target(target=target,
                       mode=dr.enums.AUTOPILOT_MODE.FULL_AUTO,
                       metric=metric,
                       worker_count=-1)
project_clf.wait_for_autopilot()<jupyter_output><empty_output><jupyter_text>2.B Anomaly detection models<jupyter_code># select anomaly detection blueprints
blueprints = project_clf.get_blueprints()
blueprints = [bp for bp in blueprints if 'anomaly' in  bp.model_type.lower()]
print(len(blueprints))
# unlock holdout
project_clf.unlock_holdout()

# train anomaly detection models
print(str(datetime.now()), 'start')
model_job_ids = []
for bp in blueprints:
    model_job_ids.append(project_clf.train(bp, sample_pct=100))
    
model_jobs = []
for i in model_job_ids:
    model_jobs.append(dr.ModelJob.get(project_clf_id, i))
    
for mj in model_jobs:
    mj.wait_for_completion()
print(str(datetime.now()), 'done')
# get the leaderboard
model_scores = get_model_scores(project_clf, metric='AUC', ascending=False)
model_scores['is_ad'] = model_scores.model_type.apply(lambda x: 'anomaly' in x.lower())
# blenders show slightly better results
model_scores.head(10)
# anomaly detection results aren't good but let's try to use them as features for supervised models
model_scores[model_scores.is_ad].head(10)<jupyter_output><empty_output><jupyter_text>2.C Supervised with anomaly detection features<jupyter_code># run anomaly detection blueprints on 64% of data to avoid the target leakage
print(str(datetime.now()), 'start')
model_job_ids = []
for bp in blueprints:
    model_job_ids.append(project_clf.train(bp, sample_pct=64.00393, scoring_type=dr.enums.SCORING_TYPE.cross_validation))
    
model_jobs = []
for i in model_job_ids:
    model_jobs.append(dr.ModelJob.get(project_clf_id, i))
    
for mj in model_jobs:
    mj.wait_for_completion()
print(str(datetime.now()), 'done')
# get the leaderboard 
model_scores = get_model_scores(project_clf, metric='AUC', ascending=False)
model_scores['is_ad'] = model_scores.model_type.apply(lambda x: 'anomaly' in x.lower())
model_scores[model_scores.is_ad & (model_scores.sample_pct < 100)].head()
# select top n models
models_n = 3
models = model_scores[model_scores.is_ad & (model_scores.sample_pct < 100)].head(models_n).model.values.tolist()
models
# get anomaly detection predictions
print(str(datetime.now()))
preds_ad = prep_ad_preds(models)
print(str(datetime.now()))
# read the origonal dataset and add anomaly detection predictions
df = pd.read_csv('data/DR_Demo_Medical_Fraud.csv')
print(df.shape)

df = df.merge(preds_ad, left_index=True, right_index=True)
print(df.shape)
# create project with additional features and the same partitioning
project_clf_ad = dr.Project.create(project_name=f'{project_name}_clf_ad', sourcedata=df)

part = dr.UserCV(user_partition_col='partition_id', 
                 cv_holdout_level=5)
project_clf_ad.set_target(target=target,
                          mode=dr.enums.AUTOPILOT_MODE.FULL_AUTO,
                          metric=metric,
                          worker_count=-1, 
                          partitioning_method=part)
project_clf_ad.wait_for_autopilot()
# get the leaderboard
model_scores_ad = get_model_scores(project_clf_ad, metric='AUC', ascending=False)
model_scores_ad.head()<jupyter_output><empty_output><jupyter_text>3. Compare results<jupyter_code>model_clf = dr.ModelRecommendation.get(project_clf.id).get_model()
model_clf_ad = dr.ModelRecommendation.get(project_clf_ad.id).get_model()
# recommended models results
print('AUC supervised                      :', model_clf.metrics['AUC']['crossValidation'])
print('AUC supervised with anomaly features:', model_clf_ad.metrics['AUC']['crossValidation'])

print('LogLoss supervised                      :', model_clf.metrics['LogLoss']['crossValidation'])
print('LogLoss supervised with anomaly features:', model_clf_ad.metrics['LogLoss']['crossValidation'])<jupyter_output>AUC supervised                      : 0.77278
AUC supervised with anomaly features: 0.765288
LogLoss supervised                      : 0.16669399999999998
LogLoss supervised with anomaly features: 0.167132<jupyter_text>4. Deployment and scoring<jupyter_code>def predict_deployment(data, datarobot_key, deployment_url, deployment_id):
    # Set HTTP headers. The charset should match the contents of the file.
    headers = {'Content-Type': 'application/json; charset=UTF-8', 'datarobot-key': datarobot_key}

    url = f'{deployment_url}/predApi/v1.0/deployments/{deployment_id}/predictions'
    
    # Make API request for predictions
    predictions_response = requests.post(
        url,
        auth=(creds['username'], creds['token']),
        data=data,
        headers=headers,
    )

    return predictions_response.json()
def predict_deployment_expl(data, datarobot_key, deployment_url, deployment_id):
    # Set HTTP headers. The charset should match the contents of the file.
    headers = {'Content-Type': 'application/json; charset=UTF-8', 'datarobot-key': datarobot_key}
    
    # prediction explanations parameters
    params = {
            'maxCodes': 3,
            'thresholdHigh': 0.1,
            'thresholdLow': 0.01,
        }
    
    url = f'{deployment_url}/predApi/v1.0/deployments/{deployment_id}/predictionExplanations'
    
    # Make API request for predictions
    predictions_response = requests.post(
        url,
        auth=(creds['username'], creds['token']),
        data=data,
        headers=headers,
        params=params
    )

    return predictions_response.json()
def flatten_json(y):
    out = {}

    def flatten(x, name=''):
        if type(x) is dict:
            for a in x:
                flatten(x[a], name + a + '_')
        elif type(x) is list:
            i = 0
            for a in x:
                flatten(a, name + str(i) + '_')
                i += 1
        else:
            out[name[:-1]] = x

    flatten(y)
    return out
# compute feature impact
feature_impacts = model_clf.get_or_request_feature_impact()

# initialize prediction explanations
pei_job = dr.PredictionExplanationsInitialization.create(project_clf.id, model_clf.id)
pei_job.wait_for_completion()
# deploy the model
deployment = dr.Deployment.create_from_learning_model(model_id=model_clf.id, 
                                                      label=f'{project_name}_clf_depl',
                                                      default_prediction_server_id=creds['pred_serv_id'])
# get prediction server url, deployment id and DataRobot key
pred_server = deployment.default_prediction_server

dr_key = pred_server['datarobot-key']
depl_url = pred_server['url']
depl_id = deployment.id
# read and prepare a dataset to score
df_scoring = pd.read_csv('data/DR_Demo_Medical_Fraud_scoring.csv')
print(df_scoring.shape)
data_to_pred = json.dumps(df_scoring.to_dict(orient='records'))
# get and precess predictions with explanations
print(str(datetime.now()))
preds_raw = predict_deployment_expl(data_to_pred, dr_key, depl_url, depl_id)

df_preds = pd.DataFrame()
for row in preds_raw['data']:
    new_row = pd.json_normalize(data=flatten_json(row))
    df_preds = pd.concat([df_preds, new_row])
print(str(datetime.now()))
cols_to_rename = {'predictionValues_0_value': 'Prediction', 
                  'predictionExplanations_0_feature': 'Primary Feature',
                  'predictionExplanations_0_featureValue': 'Primary Feature Value',
                  'predictionExplanations_0_qualitativeStrength':'Primary Feature Strength',
                  
                  'predictionExplanations_1_feature': 'Secondary Feature',
                  'predictionExplanations_1_featureValue': 'Secondary Feature Value',
                  'predictionExplanations_1_qualitativeStrength':'Secondary Feature Strength',
                  
                  'predictionExplanations_2_feature': 'Tertiary Feature',
                  'predictionExplanations_2_featureValue': 'Tertiary Feature Value',
                  'predictionExplanations_2_qualitativeStrength':'Tertiary Feature Strength'}

df_preds.rename(columns=cols_to_rename, inplace=True)
print(df_scoring.shape, df_preds.shape)
cols_to_add = ['rowId', 'Prediction', 
               'Primary Feature Value', 'Primary Feature Strength', 'Primary Feature', 
               'Secondary Feature Value', 'Secondary Feature Strength', 'Secondary Feature',
               'Tertiary Feature Value', 'Tertiary Feature Strength', 'Tertiary Feature',
              ]
df_scoring = df_scoring.merge(df_preds[cols_to_add], on='rowId')
df_scoring['Prediction Category'] = df_scoring.Prediction.apply(lambda x:
                                                                'High' if x >= 0.2 else
                                                                'Medium' if x >= 0.1 else 'Low')
print(df_scoring.shape)
# save results to use for a dashboard preparation
df_scoring.to_csv('data/DR_Demo_Medical_Fraud_scoring_predictions.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1045289962604618, 'fraction_numerical': 0.008547515878197898, 'mean_word_length': 3.756634669678148, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '334245', 'license': 'Apache-2.0', 'n_tokens_mistral': 5433, 'n_tokens_neox': 5147, 'n_words': 1201, 'path': 'Classification/Python/predicting_fraud_medical_claims/src/Predicting Fraud Medical Claims.ipynb', 'repo_name': 'DalilaB000/tutorials-for-data-scientists', 'size': '16866'}