<jupyter_start><jupyter_text>Working with Watson Machine Learning - Load Historica data to emulate a production system  Contents- [1.0 Install Python Packages](setup)- [2.0 Configure Credentials](credentials)- [3.0 OpenScale configuration](openscale)- [4.0 Get Subscriptions](subscription)- [5.0 Historical Data](historical)  1.0 Install Python Packages<jupyter_code>!rm -rf /home/spark/shared/user-libs/python3.6*

!pip install --upgrade ibm-ai-openscale==2.2.1 --no-cache | tail -n 1
!pip install --upgrade watson-machine-learning-client-V4==1.0.110 | tail -n 1
!pip install --upgrade pyspark==2.3 | tail -n 1<jupyter_output>[31mERROR: Could not install packages due to an EnvironmentError: [Errno 30] Read-only file system: 'INSTALLER'
[0m
    Uninstalling ibm-ai-openscale-2.1.21:
Requirement already satisfied, skipping upgrade: docutils<0.16,>=0.10 in /opt/conda/envs/Python-3.6-WMLCE/lib/python3.6/site-packages (from ibm-cos-sdk-core==2.6.0->ibm-cos-sdk==2.6.0->watson-machine-learning-client-V4==1.0.110) (0.14)
Requirement already satisfied, skipping upgrade: py4j==0.10.6 in /opt/conda/envs/Python-3.6-WMLCE/lib/python3.6/site-packages (from pyspark==2.3) (0.10.6)<jupyter_text>Action: restart the kernel!  2.0 Configure credentials<jupyter_code>import warnings
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>The url for `WOS_CREDENTIALS` is the url of the CP4D cluster, i.e. `https://zen-cpd-zen.apps.com`. `username` and `password` must be ones that have cluster admin privileges.<jupyter_code>WOS_CREDENTIALS = {
    "url": "https://zen1-cpd-zen1.dal10-<PASSWORD>",
    "username": "scottda-noadmin",
    "password": "<<PASSWORD>>"
}
WML_CREDENTIALS = WOS_CREDENTIALS.copy()
WML_CREDENTIALS['instance_id']='openshift'
WML_CREDENTIALS['version']='3.0.0'
%store -r MODEL_NAME
%store -r DEPLOYMENT_NAME
%store -r DEFAULT_SPACE
%store -r model_uid
%store -r binding_uid
%store -r deployment_uid
MODEL_NAME = "sda-model-8-3-2020"
DEPLOYMENT_NAME = "scottda-model-deployment-8-3-2020"
DEFAULT_SPACE = "3f2c0a53-e1bb-4e09-957d-d7b84c16a56f"<jupyter_output><empty_output><jupyter_text>3.0 Configure OpenScale  The notebook will now import the necessary libraries and configure OpenScale<jupyter_code>from watson_machine_learning_client import WatsonMachineLearningAPIClient
import json

wml_client = WatsonMachineLearningAPIClient(WML_CREDENTIALS)
from ibm_ai_openscale import APIClient4ICP
from ibm_ai_openscale.engines import *
from ibm_ai_openscale.utils import *
from ibm_ai_openscale.supporting_classes import PayloadRecord, Feature
from ibm_ai_openscale.supporting_classes.enums import *
ai_client = APIClient4ICP(WOS_CREDENTIALS)
ai_client.version<jupyter_output><empty_output><jupyter_text>4.0 Get Subscription<jupyter_code>subscription = None

if subscription is None:
    subscriptions_uids = ai_client.data_mart.subscriptions.get_uids()
    for sub in subscriptions_uids:
        if ai_client.data_mart.subscriptions.get_details(sub)['entity']['asset']['name'] == MODEL_NAME:
            print("Found existing subscription.")
            subscription = ai_client.data_mart.subscriptions.get(sub)
            subscription_uid = sub
            print("subscription_uid: " + sub)
if subscription is None:
    print("No subscription found. Please run openscale-initial-setup.ipynb to configure.")<jupyter_output>Found existing subscription.
subscription_uid: f978ce05-e4f6-42b0-81b2-be06884cd358<jupyter_text>Set Deployment UID<jupyter_code>wml_client.set.default_space(DEFAULT_SPACE)
wml_deployments = wml_client.deployments.get_details()
deployment_uid = None
for deployment in wml_deployments['resources']:
    print(deployment['entity']['name'])
    if DEPLOYMENT_NAME == deployment['entity']['name']:
        deployment_uid = deployment['metadata']['guid']
        break
        
print(deployment_uid)
binding_uid = None

binding_uid = ai_client.data_mart.bindings.get_details()['service_bindings'][0]['metadata']['guid']
if binding_uid is None:
    print("Adding binding:")
    binding_uid = ai_client.data_mart.bindings.add('WML instance', WatsonMachineLearningInstance4ICP(wml_credentials=WML_CREDENTIALS))
    bindings_details = ai_client.data_mart.bindings.get_details()
else:
    print("Found binding:")
binding_uid
wml_models = wml_client.repository.get_model_details()
model_uid = None

for model_in in wml_models['resources']:
    if MODEL_NAME == model_in['entity']['name']:
        model_uid = model_in['metadata']['guid']
        break
        
print(model_uid)<jupyter_output>5b2016f1-ad4e-47f5-af86-32497f372383<jupyter_text>8.0 Historical data    8.1 Insert historical payloads The next section of the notebook downloads and writes historical data to the payload and measurement tables to simulate a production model that has been monitored and receiving regular traffic for the last seven days. This historical data can be viewed in the Watson OpenScale user interface. The code uses the Python and REST APIs to write this data.<jupyter_code>!rm payload_history_*.json
!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/payload_history_0.json
!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/payload_history_1.json
!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/payload_history_2.json
!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/payload_history_3.json
!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/payload_history_4.json
!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/payload_history_5.json
!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/payload_history_6.json
!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/payload_history_7.json
!rm history_payloads_with_transaction_*.json
!wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/fastpath/history_payloads_with_transaction_id_0.json
!wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/fastpath/history_payloads_with_transaction_id_1.json
!wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/fastpath/history_payloads_with_transaction_id_2.json
!wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/fastpath/history_payloads_with_transaction_id_3.json
!wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/fastpath/history_payloads_with_transaction_id_4.json
!wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/fastpath/history_payloads_with_transaction_id_5.json
!wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/fastpath/history_payloads_with_transaction_id_6.json

from ibm_ai_openscale.utils.inject_demo_data import DemoData
import os

historicalData = DemoData(WOS_CREDENTIALS, ai_client)
historical_data_path=os.getcwd()

historicalData.load_historical_scoring_payload(subscription, deployment_uid,file_path=historical_data_path, day_template="history_payloads_with_transaction_id_{}.json" )
historyDays = 7
from ibm_ai_openscale.supporting_classes import PayloadRecord, Feature
import datetime
import time

for day in range(historyDays):
    print('Loading day {}'.format(day + 1))
    history_file = 'payload_history_' + str(day) + '.json'
    with open(history_file) as f:
        payloads = json.load(f)
        hourly_records = int(len(payloads) / 24)
        index = 0
        for hour in range(24):
            recordsList = []
            for i in range(hourly_records):
                score_time = str(datetime.datetime.utcnow() + datetime.timedelta(hours=(-(24*day + hour + 1))))
                recordsList.append(PayloadRecord(request=payloads[index]['request'], response=payloads[index]['response'], scoring_timestamp=score_time))
                index += 1
            subscription.payload_logging.store(records=recordsList)
print('Finished')

data_mart_id = subscription.get_details()['metadata']['url'].split('/service_bindings')[0].split('marts/')[1]
print(data_mart_id)
performance_metrics_url = WOS_CREDENTIALS['url'] + subscription.get_details()['metadata']['url'].split('/service_bindings')[0] + '/metrics'
print(performance_metrics_url)<jupyter_output>https://zen1-cpd-zen1.aida-cpd3-dal10-b3c32x128-f2c6cdc6801be85fd188b09d006f13e3-0001.us-south.containers.appdomain.cloud/v1/data_marts/00000000-0000-0000-0000-000000000000/metrics<jupyter_text>8.2 Insert historical fairness metrics<jupyter_code>!rm fairness_history*
!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/fairness_history.json
!rm history_fairness.json
!wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/history_fairness.json
import requests
from requests.auth import HTTPBasicAuth

def create_token():
    header = {
                    "Content-Type": "application/x-www-form-urlencoded",
                    "Accept": "application/json"
    }

    response = requests.Session().get(
            WOS_CREDENTIALS['url'] + '/v1/preauth/validateAuth',
            headers=header,
            auth=HTTPBasicAuth(
                WOS_CREDENTIALS['username'],
                WOS_CREDENTIALS['password']
            ),
            verify=False)

    response = handle_response(200, 'access token', response, True)
    token = response['accessToken']

    return token
iam_token = create_token()
iam_headers = {
    'Content-Type': 'application/json',
    'Authorization': 'Bearer %s' % iam_token
}

with open('history_fairness.json', 'r') as history_file:
    payloads = json.load(history_file)

for day in range(historyDays):
    print('Loading day', day + 1)
    for hour in range(24):
        score_time = (datetime.datetime.utcnow() + datetime.timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')
        index = (day * 24 + hour) % len(payloads) # wrap around and reuse values if needed
        
        qualityMetric = {
            'metric_type': 'fairness',
            'binding_id': binding_uid,
            'timestamp': score_time,
            'subscription_id': model_uid,
            'asset_revision': model_uid,
            'deployment_id': deployment_uid,
            'value': payloads[index]
        }

        response = requests.post(performance_metrics_url, json=[qualityMetric], headers=iam_headers, verify=False)

print('Finished')<jupyter_output>Loading day 1
Loading day 2
Loading day 3
Loading day 4
Loading day 5
Loading day 6
Loading day 7
Finished<jupyter_text>8.3 Insert historical debias metrics<jupyter_code>!rm history_debias.json
!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale//history_debias.json
!rm history_debias.json
!wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/history_debias.json
iam_token = create_token()
iam_headers = {
    'Content-Type': 'application/json',
    'Authorization': 'Bearer %s' % iam_token
}

with open('history_debias.json', 'r') as history_file:
    payloads = json.load(history_file)

for day in range(historyDays):
    print('Loading day', day + 1)
    for hour in range(24):
        score_time = (datetime.datetime.utcnow() + datetime.timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')
        index = (day * 24 + hour) % len(payloads) # wrap around and reuse values if needed

        qualityMetric = {
            'metric_type': 'debiased_fairness',
            'binding_id': binding_uid,
            'timestamp': score_time,
            'subscription_id': model_uid,
            'asset_revision': model_uid,
            'deployment_id': deployment_uid,
            'value': payloads[index]
        }

        response = requests.post(performance_metrics_url, json=[qualityMetric], headers=iam_headers, verify=False)
print('Finished')<jupyter_output>Loading day 1
Loading day 2
Loading day 3
Loading day 4
Loading day 5
Loading day 6
Loading day 7
Finished<jupyter_text>8.4 Insert historical quality metrics<jupyter_code>iam_token = create_token()
iam_headers = {
    'Content-Type': 'application/json',
    'Authorization': 'Bearer %s' % iam_token
}

measurements = [0.76, 0.78, 0.68, 0.72, 0.73, 0.77, 0.80]
for day in range(historyDays):
    print('Day', day + 1)
    for hour in range(24):
        score_time = (datetime.datetime.utcnow() + datetime.timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')
        qualityMetric = {
            'metric_type': 'quality',
            'binding_id': binding_uid,
            'timestamp': score_time,
            'subscription_id': model_uid,
            'asset_revision': model_uid,
            'deployment_id': deployment_uid,
            'value': {
                'quality': measurements[day],
                'threshold': 0.7,
                'metrics': [
                    {
                        'name': 'auroc',
                        'value': measurements[day],
                        'threshold': 0.7
                    }
                ]
            }
        }

        response = requests.post(performance_metrics_url, json=[qualityMetric], headers=iam_headers, verify=False)
print('Finished')<jupyter_output>Day 1
Day 2
Day 3
Day 4
Day 5
Day 6
Day 7
Finished<jupyter_text>8.5 Insert historical confusion matrixes<jupyter_code>!rm history_quality_metrics.json
!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_quality_metrics.json
!rm history_quality_metrics.json
!wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wos/history_quality_metrics.json
measurements_url = WOS_CREDENTIALS['url'] + subscription.get_details()['metadata']['url'].split('/service_bindings')[0] + '/measurements'
print(measurements_url)
with open('history_quality_metrics.json') as json_file:
    records = json.load(json_file)

for day in range(historyDays):
    index = 0
    measurments = []
    print('Day', day + 1)
    
    for hour in range(24):
        score_time = (datetime.datetime.utcnow() + datetime.timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')

        measurement = {
            "monitor_definition_id": 'quality',
            "binding_id": subscription.binding_uid,
            "subscription_id": subscription.uid,
            "asset_id": subscription.source_uid,
            'metrics': [records[index]['metrics']],
            'sources': [records[index]['sources']],
            'timestamp': score_time
        }

        measurments.append(measurement)
        index+=1

    response = requests.post(measurements_url, json=measurments, headers=ai_client._get_headers(), verify=False)

print('Finished')<jupyter_output>Day 1
Day 2
Day 3
Day 4
Day 5
Day 6
Day 7
Finished<jupyter_text>8.6 Insert historical performance metrics<jupyter_code>iam_token = create_token()
iam_headers = {
    'Content-Type': 'application/json',
    'Authorization': 'Bearer %s' % iam_token
}

for day in range(historyDays):
    print('Day', day + 1)
    for hour in range(24):
        score_time = (datetime.datetime.utcnow() + datetime.timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')
        score_count = random.randint(60, 600)
        score_resp = random.uniform(60, 300)

        performanceMetric = {
            'metric_type': 'performance',
            'binding_id': binding_uid,
            'timestamp': score_time,
            'subscription_id': model_uid,
            'asset_revision': model_uid,
            'deployment_id': deployment_uid,
            'value': {
                'response_time': score_resp,
                'records': score_count
            }
        }

        response = requests.post(performance_metrics_url, json=[performanceMetric], headers=iam_headers, verify=False)
print('Finished')<jupyter_output>Day 1
Day 2
Day 3
Day 4
Day 5
Day 6
Day 7
Finished<jupyter_text>8.7 Insert historical manual labeling<jupyter_code>manual_labeling_url = WOS_CREDENTIALS['url'] + subscription.get_details()['metadata']['url'].split('/service_bindings')[0] + '/manual_labelings'
print(manual_labeling_url)
!rm history_manual_labeling.json
!wget https://raw.githubusercontent.com/IBM/credit-risk-workshop-cpd/master/data/openscale/history_manual_labeling.json
!rm history_manual_labeling.json
!wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/history_manual_labeling.json
iam_token = create_token()
iam_headers = {
    'Content-Type': 'application/json',
    'Authorization': 'Bearer %s' % iam_token
}

with open('history_manual_labeling.json', 'r') as history_file:
    records = json.load(history_file)

for day in range(historyDays):
    print('Loading day', day + 1)
    record_json = []
    for hour in range(24):
        for record in records:
            if record['fastpath_history_day'] == day and record['fastpath_history_hour'] == hour:
                record['binding_id'] = binding_uid
                record['subscription_id'] = model_uid
                record['asset_revision'] = model_uid
                record['deployment_id'] = deployment_uid
                record['scoring_timestamp'] = (datetime.datetime.utcnow() + datetime.timedelta(hours=(-(24*day + hour + 1)))).strftime('%Y-%m-%dT%H:%M:%SZ')
                record_json.append(record)
    response = requests.post(manual_labeling_url, json=record_json, headers=iam_headers, verify=False)

print('Finished')<jupyter_output>Loading day 1
Loading day 2
Loading day 3
Loading day 4
Loading day 5
Loading day 6
Loading day 7
Finished<jupyter_text>8.8 Additional data to help debugging<jupyter_code>print('Datamart:', data_mart_id)
print('Model:', model_uid)
print('Deployment:', deployment_uid)
print('Binding:', binding_uid)
# print('Scoring URL:', credit_risk_scoring_endpoint)<jupyter_output>Datamart: 00000000-0000-0000-0000-000000000000
Model: 5b2016f1-ad4e-47f5-af86-32497f372383
Deployment: d435943c-d802-4e3b-b34c-e0f42fa4ee52
Binding: 405e817b-3715-453b-91a7-6a36186de087<jupyter_text>8.9 Identify transactions for Explainability Transaction IDs identified by the cells below can be copied and pasted into the Explainability tab of the OpenScale dashboard.<jupyter_code>payload_data = subscription.payload_logging.get_table_content(limit=10)
payload_data.filter(items=['scoring_id', 'predictedLabel', 'probability'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12053313742981582, 'fraction_numerical': 0.026866768116702525, 'mean_word_length': 4.3294183445190155, 'pattern_counts': {'":': 9, '<': 54, '<?xml version=': 0, '>': 56, 'https://': 26, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 4, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '206234', 'license': 'Apache-2.0', 'n_tokens_mistral': 6704, 'n_tokens_neox': 6225, 'n_words': 1075, 'path': 'notebooks/with-output/openscale-historic-data-with-output.ipynb', 'repo_name': 'sakanmfec/cloudpakfordata_workshop', 'size': '19104'}