<jupyter_start><jupyter_text>This notebook should be run using Default Spark 3.0 & Python 3.8 runtime environment. If you are viewing this in Watson Studio and do not see Python 3.8.x with Spark in the upper right corner of your screen, please update the runtime now. It requires service credentials for the following services:Watson OpenScaleWatson Machine LearningDb2  Credentials for IBM Cloud Pak for DataIn the following code boxes, you need to replace the sample data with your own credentials. You can acquire the information from your system administrator or through the Cloud Pak for Data dashboard.  Obtaining your Watson OpenScale credentialsYou can retrieve the URL by running the following command: `oc get route -n namespace1 --no-headers | awk '{print $2}'`Replace the `namespace1` variable with your namespace.You should have been assigned a username and password when you were added to the Cloud Pak for Data system. You might need to ask either your database administrator or your system administrator for some of the information.<jupyter_code>############################################################################################
# Paste your Watson OpenScale credentials into the following section and then run this cell.
############################################################################################
WOS_CREDENTIALS = {
      "url": "<CP4D cluster URL>",
      "username": "*****",
      "password": "*****"
  }<jupyter_output><empty_output><jupyter_text>Your Watson OpenScale GUIDFor most systems, the default GUID is already entered for you. You would only need to update this particular entry if the GUID was changed from the default.<jupyter_code>WOS_GUID="00000000-0000-0000-0000-000000000000"<jupyter_output><empty_output><jupyter_text>Your Watson OpenScale GUIDFor most systems, the default GUID is already entered for you. You would only need to update this particular entry if the GUID was changed from the default.<jupyter_code>WML_CREDENTIALS = WOS_CREDENTIALS.copy()
WML_CREDENTIALS['instance_id']='openshift'
WML_CREDENTIALS['version']='4.0' #If your env is CP4D 4.0 then specify "4.0" instead of "3.5"
WML_CREDENTIALS<jupyter_output><empty_output><jupyter_text>Your database credentials and schemaNormally, you must obtain the database credentials and schema name from your database administrator, however, if you deployed one of the database options from the Cloud Pak for Data UI, you should be able to retrieve credentials yourself by going to My data and right-clicking on the database tile.<jupyter_code>####################################################################################
# Paste your database credentials into the following section and then run this cell.
####################################################################################
DATABASE_CREDENTIALS = {
      "db": "SAMPLE",
      "db_type": "db2",
      "hostname": "<host>",
      "password": "******",
      "port": "<port>",
      "username": "*****"
  }
SCHEMA_NAME = "<schema_name>"<jupyter_output><empty_output><jupyter_text>Integration system credentials (IBM OpenPages MRG) As part of the closed beta, you were sent an email message from IBM that provides the URL, username, and password of an IBM OpenPages system that you can use. Replace the following credentials with the ones that you received in that email message.<jupyter_code>#########################################################################################
# Paste your IBM OpenPages credentials into the following section and then run this cell.
#########################################################################################
OPENPAGES_CREDENTIALS = {
     "url": "<Openpages URL>",
     "username": "*****",
     "password": "*****"
  }<jupyter_output><empty_output><jupyter_text>Model IDFor the model that you create, copy the `model_id` from the OpenPages Model URL from the browser address bar. For example, if you create a model and view it in the OpenPages UI, you might see a URL such as "https://mrgbeta.op-ibm.com/app/jspview/react/grc/task-view/7916?". In this example, the `model_id` is **7916**. Paste that value into the following code cell.<jupyter_code>####################################################################
# Paste your model ID in the following field and then run this cell.
####################################################################
openpages_model_id="<model_id>"<jupyter_output><empty_output><jupyter_text>Package installationThe following opensource packages must be installed into this notebook instance so that they are available to use during processing.<jupyter_code>!rm -rf /home/spark/shared/user-libs/python3.6*

!pip install --upgrade pyspark==3.0.3  --no-cache | tail -n 1
!pip install numpy==1.20.1 --no-cache | tail -n 1
!pip install scikit-learn==0.23.1 --no-cache | tail -n 1
!pip install --upgrade pandas==1.2.3 --no-cache | tail -n 1
!pip install --upgrade SciPy --no-cache | tail -n 1
!pip install --upgrade ibm-watson-machine-learning --user | tail -n 1
!pip install --upgrade ibm_watson_openscale --no-cache | tail -n 1<jupyter_output><empty_output><jupyter_text>Load the training data from GithubSo you don't have to manually generate training data, we've provided a sample and placed it in a publicly available Github repo.<jupyter_code>!rm german_credit_data_biased_training.csv
!wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/german_credit_data_biased_training.csv
import json
import requests
import base64
from requests.auth import HTTPBasicAuth
import time<jupyter_output><empty_output><jupyter_text>Deploy the modelsThe following cells deploy both the pre-prod and challenger models into the Watson Machine Learning instance.<jupyter_code>PRE_PROD_MODEL_NAME="German Credit Risk Model - PreProd"
PRE_PROD_DEPLOYMENT_NAME="German Credit Risk Model - PreProd"

PRE_PROD_CHALLENGER_MODEL_NAME="German Credit Risk Model - Challenger"
PRE_PROD_CHALLENGER_DEPLOYMENT_NAME="German Credit Risk Model - Challenger"

PRE_PROD_SPACE_NAME="wml_preprod"<jupyter_output><empty_output><jupyter_text>Initialize WML client and set the space_id parameterAnalytic spaces are used to categorize assets and can be associated with a project inside of Watson Studio.<jupyter_code>from ibm_watson_machine_learning import APIClient
wml_client = APIClient(WML_CREDENTIALS)
print(wml_client.service_instance.get_url())

wml_client.spaces.list()

# Find and set the default space
space_name=PRE_PROD_SPACE_NAME
spaces = wml_client.spaces.get_details()['resources']
space_id = None
for space in spaces:
    if space['entity']['name'] == space_name:
        space_id = space["metadata"]["id"]
        print(space_id)
if space_id is None:
    space_id = wml_client.spaces.store(
        meta_props={wml_client.spaces.ConfigurationMetaNames.NAME: space_name})["metadata"]["id"]

wml_client.set.default_space(space_id)


wml_client.version<jupyter_output><empty_output><jupyter_text>Deploy the Spark Credit Risk Model to WMLThe following cell deploys the Spark version of the German Credit Risk Model to the specified Machine Learning instance in the specified deployment space. You'll notice that this version of the German Credit Risk model has an auc-roc score around 71%.<jupyter_code>import numpy 
numpy.version.version

import pandas as pd
import json

from pyspark import SparkContext, SQLContext
from pyspark.ml import Pipeline
from pyspark.ml.classification import RandomForestClassifier,GBTClassifier
from pyspark.ml.evaluation import BinaryClassificationEvaluator
from pyspark.ml.feature import StringIndexer, VectorAssembler, IndexToString
from pyspark.sql.types import StructType, DoubleType, StringType, ArrayType

from pyspark.sql import SparkSession
from pyspark import SparkFiles

spark = SparkSession.builder.getOrCreate()
pd_data = pd.read_csv("german_credit_data_biased_training.csv", sep=",", header=0)
spark_df = spark.read.csv(path="german_credit_data_biased_training.csv", sep=",", header=True, inferSchema=True)
spark_df.head()

(train_data, test_data) = spark_df.randomSplit([0.9, 0.1], 24)
print("Number of records for training: " + str(train_data.count()))
print("Number of records for evaluation: " + str(test_data.count()))

si_CheckingStatus = StringIndexer(inputCol='CheckingStatus', outputCol='CheckingStatus_IX')
si_CreditHistory = StringIndexer(inputCol='CreditHistory', outputCol='CreditHistory_IX')
si_LoanPurpose = StringIndexer(inputCol='LoanPurpose', outputCol='LoanPurpose_IX')
si_ExistingSavings = StringIndexer(inputCol='ExistingSavings', outputCol='ExistingSavings_IX')
si_EmploymentDuration = StringIndexer(inputCol='EmploymentDuration', outputCol='EmploymentDuration_IX')
si_Sex = StringIndexer(inputCol='Sex', outputCol='Sex_IX')
si_OthersOnLoan = StringIndexer(inputCol='OthersOnLoan', outputCol='OthersOnLoan_IX')
si_OwnsProperty = StringIndexer(inputCol='OwnsProperty', outputCol='OwnsProperty_IX')
si_InstallmentPlans = StringIndexer(inputCol='InstallmentPlans', outputCol='InstallmentPlans_IX')
si_Housing = StringIndexer(inputCol='Housing', outputCol='Housing_IX')
si_Job = StringIndexer(inputCol='Job', outputCol='Job_IX')
si_Telephone = StringIndexer(inputCol='Telephone', outputCol='Telephone_IX')
si_ForeignWorker = StringIndexer(inputCol='ForeignWorker', outputCol='ForeignWorker_IX')
si_Label = StringIndexer(inputCol="Risk", outputCol="label").fit(spark_df)
label_converter = IndexToString(inputCol="prediction", outputCol="predictedLabel", labels=si_Label.labels)

va_features = VectorAssembler(
inputCols=["CheckingStatus_IX", "CreditHistory_IX", "LoanPurpose_IX", "ExistingSavings_IX",
           "EmploymentDuration_IX", "Sex_IX", "OthersOnLoan_IX", "OwnsProperty_IX", "InstallmentPlans_IX",
           "Housing_IX", "Job_IX", "Telephone_IX", "ForeignWorker_IX", "LoanDuration", "LoanAmount",
           "InstallmentPercent", "CurrentResidenceDuration", "LoanDuration", "Age", "ExistingCreditsCount",
           "Dependents"], outputCol="features")

classifier=GBTClassifier(featuresCol="features")

pipeline = Pipeline(
stages=[si_CheckingStatus, si_CreditHistory, si_EmploymentDuration, si_ExistingSavings, si_ForeignWorker,
        si_Housing, si_InstallmentPlans, si_Job, si_LoanPurpose, si_OthersOnLoan,
        si_OwnsProperty, si_Sex, si_Telephone, si_Label, va_features, classifier, label_converter])

model = pipeline.fit(train_data)
predictions = model.transform(test_data)
evaluator = BinaryClassificationEvaluator(rawPredictionCol="prediction")
auc = evaluator.evaluate(predictions)

print("Accuracy = %g" % auc)

# Remove existing model and deployment
MODEL_NAME=PRE_PROD_MODEL_NAME
DEPLOYMENT_NAME=PRE_PROD_DEPLOYMENT_NAME

deployment_details = wml_client.deployments.get_details()
for deployment in deployment_details['resources']:
    deployment_id = wml_client.deployments.get_id(deployment)
    model_id = deployment['entity']['asset']['id']
    if deployment['entity']['name'] == DEPLOYMENT_NAME:
        print('Deleting deployment id', deployment_id)
        wml_client.deployments.delete(deployment_id)
        print('Deleting model id', model_id)
        wml_client.repository.delete(model_id)
wml_client.repository.list_models()

# Save Model
software_spec_uid = wml_client.software_specifications.get_id_by_name("spark-mllib_3.0")
model_props_rf = {
    wml_client.repository.ModelMetaNames.NAME: MODEL_NAME,
    wml_client.repository.ModelMetaNames.DESCRIPTION: MODEL_NAME,
    wml_client.repository.ModelMetaNames.SOFTWARE_SPEC_UID: software_spec_uid,
    wml_client.repository.ModelMetaNames.TYPE: 'mllib_3.0'
}
print(model_props_rf)
published_model_details = wml_client.repository.store_model(model=model, meta_props=model_props_rf, training_data=train_data, pipeline=pipeline)
print(published_model_details)

# List models in the repository
wml_client.repository.list_models()

# Get the model UID
pre_prod_model_uid = wml_client.repository.get_model_uid(published_model_details)
pre_prod_model_uid


# Deploy model
wml_deployments = wml_client.deployments.get_details()
pre_prod_deployment_uid = None
for deployment in wml_deployments['resources']:
    if DEPLOYMENT_NAME == deployment['entity']['name']:
        pre_prod_deployment_uid = wml_client.deployments.get_id(deployment)
        break

if pre_prod_deployment_uid is None:
    print("Deploying model...")
    meta_props = {
        wml_client.deployments.ConfigurationMetaNames.NAME: DEPLOYMENT_NAME,
        wml_client.deployments.ConfigurationMetaNames.DESCRIPTION: DEPLOYMENT_NAME,
        wml_client.deployments.ConfigurationMetaNames.ONLINE: {}
    }
    deployment = wml_client.deployments.create(artifact_uid=pre_prod_model_uid, name=DEPLOYMENT_NAME, meta_props=meta_props)
    pre_prod_deployment_uid = wml_client.deployments.get_id(deployment)

print("Model id: {}".format(pre_prod_model_uid))
print("Deployment id: {}".format(pre_prod_deployment_uid))

pre_prod_deployment_uid=wml_client.deployments.get_id(deployment)
pre_prod_deployment_uid

fields = ["CheckingStatus","LoanDuration","CreditHistory","LoanPurpose","LoanAmount","ExistingSavings","EmploymentDuration","InstallmentPercent","Sex","OthersOnLoan","CurrentResidenceDuration","OwnsProperty","Age","InstallmentPlans","Housing","ExistingCreditsCount","Job","Dependents","Telephone","ForeignWorker"]
values = [
  ["no_checking",13,"credits_paid_to_date","car_new",1343,"100_to_500","1_to_4",2,"female","none",3,"savings_insurance",46,"none","own",2,"skilled",1,"none","yes"],
  ["no_checking",24,"prior_payments_delayed","furniture",4567,"500_to_1000","1_to_4",4,"male","none",4,"savings_insurance",36,"none","free",2,"management_self-employed",1,"none","yes"],
  ["0_to_200",26,"all_credits_paid_back","car_new",863,"less_100","less_1",2,"female","co-applicant",2,"real_estate",38,"none","own",1,"skilled",1,"none","yes"],
  ["0_to_200",14,"no_credits","car_new",2368,"less_100","1_to_4",3,"female","none",3,"real_estate",29,"none","own",1,"skilled",1,"none","yes"],
  ["0_to_200",4,"no_credits","car_new",250,"less_100","unemployed",2,"female","none",3,"real_estate",23,"none","rent",1,"management_self-employed",1,"none","yes"],
  ["no_checking",17,"credits_paid_to_date","car_new",832,"100_to_500","1_to_4",2,"male","none",2,"real_estate",42,"none","own",1,"skilled",1,"none","yes"],
  ["no_checking",33,"outstanding_credit","appliances",5696,"unknown","greater_7",4,"male","co-applicant",4,"unknown",54,"none","free",2,"skilled",1,"yes","yes"],
  ["0_to_200",13,"prior_payments_delayed","retraining",1375,"100_to_500","4_to_7",3,"male","none",3,"real_estate",37,"none","own",2,"management_self-employed",1,"none","yes"]
]

payload_scoring = {"fields": fields,"values": values}
payload = {
    wml_client.deployments.ScoringMetaNames.INPUT_DATA: [payload_scoring]
}
scoring_response = wml_client.deployments.score(pre_prod_deployment_uid, payload)

scoring_response<jupyter_output><empty_output><jupyter_text>Deploy the Scikit-Learn Credit Risk Model to Watson Machine LearningThe following cell deploys the Scikit-learn version of the German Credit Risk Model to the specified Machine Learning instance in the specified deployment space. This version of the German Credit Risk model has an auc-roc score around 85% and will be called the "Challenger."<jupyter_code>import sklearn
sklearn.__version__
import pandas as pd
import json
import sys
import numpy
import sklearn
import sklearn.ensemble
numpy.set_printoptions(threshold=sys.maxsize)
from sklearn.ensemble.gradient_boosting import GradientBoostingClassifier
from sklearn.utils.multiclass import type_of_target
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OrdinalEncoder
from sklearn.compose import ColumnTransformer
from sklearn.model_selection import cross_validate
from sklearn.metrics import get_scorer
from sklearn.model_selection import cross_validate
from sklearn.metrics import classification_report

data_df=pd.read_csv ("german_credit_data_biased_training.csv")

data_df.head()

target_label_name = "Risk"
feature_cols= data_df.drop(columns=[target_label_name])
label= data_df[target_label_name]

# Set model evaluation properties
optimization_metric = 'roc_auc'
random_state = 33
cv_num_folds = 3
holdout_fraction = 0.1

if type_of_target(label.values) in ['multiclass', 'binary']:
    X_train, X_holdout, y_train, y_holdout = train_test_split(feature_cols, label, test_size=holdout_fraction, random_state=random_state, stratify=label.values)
else:
    X_train, X_holdout, y_train, y_holdout = train_test_split(feature_cols, label, test_size=holdout_fraction, random_state=random_state)

# Data preprocessing transformer generation

numeric_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())])
categorical_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('OrdinalEncoder', OrdinalEncoder(categories='auto',dtype=numpy.float64 ))])

numeric_features = feature_cols.select_dtypes(include=['int64', 'float64']).columns
categorical_features = feature_cols.select_dtypes(include=['object']).columns

preprocessor = ColumnTransformer(
    transformers=[
        ('num', numeric_transformer, numeric_features),
        ('cat', categorical_transformer, categorical_features)])

# Initiate model and create pipeline
model=GradientBoostingClassifier()
gbt_pipeline = Pipeline(steps=[('preprocessor', preprocessor), ('classifier', model)])
model_gbt=gbt_pipeline.fit(X_train, y_train)

y_pred = model_gbt.predict(X_holdout)


# Evaluate model performance on test data and Cross validation
scorer = get_scorer(optimization_metric)
scorer(model_gbt,X_holdout, y_holdout)

# Cross validation -3 folds
cv_results = cross_validate(model_gbt,X_train,y_train, scoring={optimization_metric:scorer})
numpy.mean(cv_results['test_' + optimization_metric])

print(classification_report(y_pred, y_holdout))


# Remove existing model and deployment
MODEL_NAME=PRE_PROD_CHALLENGER_MODEL_NAME
DEPLOYMENT_NAME=PRE_PROD_CHALLENGER_DEPLOYMENT_NAME

deployment_details = wml_client.deployments.get_details()
for deployment in deployment_details['resources']:
    deployment_id = wml_client.deployments.get_id(deployment)
    model_id = deployment['entity']['asset']['id']
    if deployment['entity']['name'] == DEPLOYMENT_NAME:
        print('Deleting deployment id', deployment_id)
        wml_client.deployments.delete(deployment_id)
        print('Deleting model id', model_id)
        wml_client.repository.delete(model_id)
wml_client.repository.list_models()

# Store Model
#Note if there is specification related exception then use "default_py3.7_opence" instead of default_py3.8
software_spec_uid = wml_client.software_specifications.get_id_by_name("default_py3.8")
model_props_gbt = {
    wml_client.repository.ModelMetaNames.NAME: MODEL_NAME,
    wml_client.repository.ModelMetaNames.DESCRIPTION: MODEL_NAME,
    wml_client.repository.ModelMetaNames.SOFTWARE_SPEC_UID: software_spec_uid,
    wml_client.repository.ModelMetaNames.TYPE: "scikit-learn_0.23"
}

published_model_details = wml_client.repository.store_model(model=model_gbt, meta_props=model_props_gbt, training_data=feature_cols,training_target=label)
print(published_model_details)

# List models in the repository
wml_client.repository.list_models()

# Get the model UID
challenger_model_uid = wml_client.repository.get_model_uid(published_model_details)
print('challenger_model_uid ', challenger_model_uid)


# Deploy model
wml_deployments = wml_client.deployments.get_details()
challenger_deployment_uid = None
for deployment in wml_deployments['resources']:
    if DEPLOYMENT_NAME == deployment['entity']['name']:
        challenger_deployment_uid = wml_client.deployments.get_id(deployment)
        break

if challenger_deployment_uid is None:
    print("Deploying model...")
    meta_props = {
        wml_client.deployments.ConfigurationMetaNames.NAME: DEPLOYMENT_NAME,
        wml_client.deployments.ConfigurationMetaNames.DESCRIPTION: DEPLOYMENT_NAME,
        wml_client.deployments.ConfigurationMetaNames.ONLINE: {}
    }
    deployment = wml_client.deployments.create(artifact_uid=challenger_model_uid, meta_props=meta_props)
    challenger_deployment_uid = wml_client.deployments.get_uid(deployment)

print("Model id: {}".format(challenger_model_uid))
print("Deployment id: {}".format(challenger_deployment_uid))

challenger_deployment_uid=wml_client.deployments.get_id(deployment)
challenger_deployment_uid


# Sample scoring
fields = ["CheckingStatus","LoanDuration","CreditHistory","LoanPurpose","LoanAmount","ExistingSavings","EmploymentDuration","InstallmentPercent","Sex","OthersOnLoan","CurrentResidenceDuration","OwnsProperty","Age","InstallmentPlans","Housing","ExistingCreditsCount","Job","Dependents","Telephone","ForeignWorker"]
values = [
  ["no_checking",13,"credits_paid_to_date","car_new",1343,"100_to_500","1_to_4",2,"female","none",3,"savings_insurance",46,"none","own",2,"skilled",1,"none","yes"],
  ["no_checking",24,"prior_payments_delayed","furniture",4567,"500_to_1000","1_to_4",4,"male","none",4,"savings_insurance",36,"none","free",2,"management_self-employed",1,"none","yes"],
  ["0_to_200",26,"all_credits_paid_back","car_new",863,"less_100","less_1",2,"female","co-applicant",2,"real_estate",38,"none","own",1,"skilled",1,"none","yes"],
  ["0_to_200",14,"no_credits","car_new",2368,"less_100","1_to_4",3,"female","none",3,"real_estate",29,"none","own",1,"skilled",1,"none","yes"],
  ["0_to_200",4,"no_credits","car_new",250,"less_100","unemployed",2,"female","none",3,"real_estate",23,"none","rent",1,"management_self-employed",1,"none","yes"],
  ["no_checking",17,"credits_paid_to_date","car_new",832,"100_to_500","1_to_4",2,"male","none",2,"real_estate",42,"none","own",1,"skilled",1,"none","yes"],
  ["no_checking",33,"outstanding_credit","appliances",5696,"unknown","greater_7",4,"male","co-applicant",4,"unknown",54,"none","free",2,"skilled",1,"yes","yes"],
  ["0_to_200",13,"prior_payments_delayed","retraining",1375,"100_to_500","4_to_7",3,"male","none",3,"real_estate",37,"none","own",2,"management_self-employed",1,"none","yes"]
]

payload_scoring = {"fields": fields,"values": values}
payload = {
    wml_client.deployments.ScoringMetaNames.INPUT_DATA: [payload_scoring]
}
scoring_response = wml_client.deployments.score(challenger_deployment_uid, payload)

scoring_response<jupyter_output><empty_output><jupyter_text>Configure OpenScale The notebook will now import the necessary libraries and set up a Python OpenScale client.<jupyter_code>from ibm_cloud_sdk_core.authenticators import CloudPakForDataAuthenticator
from ibm_watson_openscale import APIClient
from ibm_watson_openscale.utils import *
from ibm_watson_openscale.supporting_classes import *
from ibm_watson_openscale.supporting_classes.enums import *
from ibm_watson_openscale.supporting_classes.payload_record import PayloadRecord
import uuid
authenticator = CloudPakForDataAuthenticator(
        url=WOS_CREDENTIALS['url'],
        username=WOS_CREDENTIALS['username'],
        password=WOS_CREDENTIALS['password'],
        disable_ssl_verification=True
    )

wos_client = APIClient(service_url=WOS_CREDENTIALS['url'],authenticator=authenticator)
wos_client.version<jupyter_output><empty_output><jupyter_text>Create schema and datamart Set up datamartWatson OpenScale uses a database to store payload logs and calculated metrics. If an OpenScale datamart exists in Db2, the existing datamart will be used and no data will be overwritten.<jupyter_code>wos_client.data_marts.show()
data_marts = wos_client.data_marts.list().result.data_marts
if len(data_marts) == 0:
    if DATABASE_CREDENTIALS is not None:
        if SCHEMA_NAME is None: 
            print("Please specify the SCHEMA_NAME and rerun the cell")

        print("Setting up external datamart")
        added_data_mart_result = wos_client.data_marts.add(
                background_mode=False,
                name="WOS Data Mart",
                description="Data Mart created by WOS tutorial notebook",
                database_configuration=DatabaseConfigurationRequest(
                  database_type=DatabaseType.DB2,
                    credentials=PrimaryStorageCredentialsLong(
                        hostname=DATABASE_CREDENTIALS["hostname"],
                        username=DATABASE_CREDENTIALS["username"],
                        password=DATABASE_CREDENTIALS["password"],
                        db=DATABASE_CREDENTIALS["db"],
                        port=DATABASE_CREDENTIALS["port"]
                    ),
                    location=LocationSchemaName(
                        schema_name= SCHEMA_NAME
                    )
                )
             ).result
    else:
        print("Setting up internal datamart")
        added_data_mart_result = wos_client.data_marts.add(
                background_mode=False,
                name="WOS Data Mart",
                description="Data Mart created by WOS tutorial notebook", 
                internal_database = True).result
        
    data_mart_id = added_data_mart_result.metadata.id
    
else:
    data_mart_id=data_marts[0].metadata.id
    print("Using existing datamart {}".format(data_mart_id))
data_mart_id<jupyter_output><empty_output><jupyter_text>Generate an ICP tokenThe following is a function that will generate an ICP access token used to interact with the Watson OpenScale APIs<jupyter_code>import requests
import urllib3
from http import HTTPStatus

def get_iamtoken(url, username, password):
    fqdn = urllib3.util.parse_url(url).netloc
    domain = '.'.join(fqdn.split('.')[1:])
    token_url = 'https://cp-console.{}/idprovider/v1/auth/identitytoken'.format(domain)
    data = {
        'grant_type': 'password',
        'username': username,
        'password': password,
        'scope': 'openid'
    }
    return requests.post(token_url, data, verify=False)

def get_accesstoken(url, username, iamtoken):
    url = '{}/v1/preauth/validateAuth'.format(url)
    headers = {
        'Content-type': 'application/json',
        'username': username,
        'iam-token': iamtoken
    }
    return requests.get(url, headers=headers, verify=False)

def generate_access_token():
    url=WOS_CREDENTIALS['url']
    username=WOS_CREDENTIALS['username']
    password=<PASSWORD>['password'] 
    response = get_iamtoken(url,username,password)
    #service is not available when iamintegration=false so fall back to old way of generating code
    if response.status_code==HTTPStatus.SERVICE_UNAVAILABLE:
        url = '{}/v1/preauth/validateAuth'.format(url)
        headers = {'Content-type': 'application/json'}
        data = {
            'grant_type': 'password',
            'username': username,
            'password': password
        }
        return requests.get(url, headers=headers, auth= (username,password),verify=False).json()['accessToken']
        
    else:
        return get_accesstoken(url,username, response.json()['access_token']).json()['accessToken']<jupyter_output><empty_output><jupyter_text>Bind WML machine learning instance as Pre-ProdWatson OpenScale needs to be bound to the Watson Machine Learning instance to capture payload data into and out of the model. If a binding with name "WML Pre-Prod" already exists, this code will delete that binding a create a new one.**Note**: Binding with name `WML Pre-Prod` is assumed to be only created by this notebook.<jupyter_code>SERVICE_PROVIDER_NAME = "WML Pre-Prod"
SERVICE_PROVIDER_DESCRIPTION = "Added by tutorial WOS notebook."
service_providers = wos_client.service_providers.list().result.service_providers
for service_provider in service_providers:
    service_instance_name = service_provider.entity.name
    if service_instance_name == SERVICE_PROVIDER_NAME:
        service_provider_id = service_provider.metadata.id
        wos_client.service_providers.delete(service_provider_id)
        print("Deleted existing service_provider for WML instance: {}".format(service_provider_id))
added_service_provider_result = wos_client.service_providers.add(
        name=SERVICE_PROVIDER_NAME,
        description=SERVICE_PROVIDER_DESCRIPTION,
        service_type=ServiceTypes.WATSON_MACHINE_LEARNING,
        deployment_space_id = space_id,
        operational_space_id = "pre_production",
        credentials=WMLCredentialsCP4D(
            url=WML_CREDENTIALS["url"],
            username=WML_CREDENTIALS["username"],
            password=<PASSWORD>["password"],
            instance_id=WML_CREDENTIALS["instance_id"]
        ),
        background_mode=False
    ).result
service_provider_id = added_service_provider_result.metadata.id
wos_client.service_providers.show()<jupyter_output><empty_output><jupyter_text>Create an integration to IBM OpenPagesTo push metrics from Watson OpenScale to IBM OpenPages, a connection must be established between the two services.<jupyter_code>headers = {}
headers["Content-Type"] = "application/json"
headers["Authorization"] = "Bearer {}".format(generate_access_token())

INTEGRATED_SYSTEMS_URL = WOS_CREDENTIALS["url"] + "/openscale/{0}/v2/integrated_systems".format(WOS_GUID)

payload = {
    "name": "OpenPages Connection",
    "type": "open_pages",
    "description": "Integration with OpenPages",
    "credentials": OPENPAGES_CREDENTIALS
}

response = requests.post(INTEGRATED_SYSTEMS_URL, json=payload, headers=headers, verify=False)
json_data = response.json()
print(json_data)

if "metadata" in json_data and "id" in json_data["metadata"]:
    integrated_system_id = json_data["metadata"]["id"]
    print(integrated_system_id)<jupyter_output><empty_output><jupyter_text>Subscriptions Remove existing PreProd and Challenger credit risk subscriptionsThis code removes previous subscriptions with name `German Credit Risk Model - PreProd` and `German Credit Risk Model - Challenger` to refresh the monitors with the new model and new data.<jupyter_code>subscriptions = wos_client.subscriptions.list().result.subscriptions
for subscription in subscriptions:
    sub_name = subscription.entity.asset.name
    if sub_name == PRE_PROD_MODEL_NAME or sub_name == PRE_PROD_CHALLENGER_MODEL_NAME:
        wos_client.subscriptions.delete(subscription.metadata.id)
        print('Deleted existing subscription for', sub_name)
wos_client.subscriptions.show()
# Create preprod subscription
pre_prod_deployment = wml_client.deployments.get_details(deployment_uid=pre_prod_deployment_uid)
pre_prod_asset_details=wos_client.service_providers.get_deployment_asset(data_mart_id=data_mart_id,service_provider_id=service_provider_id,deployment_id=pre_prod_deployment_uid,deployment_space_id=space_id)
pre_prod_asset = Asset(
    asset_id=pre_prod_asset_details["entity"]["asset"]["asset_id"],
    name=pre_prod_asset_details["entity"]["asset"]["name"],
    url=pre_prod_asset_details["entity"]["asset"]["url"],
    asset_type=AssetTypes.MODEL,
    input_data_type=InputDataType.STRUCTURED,
    problem_type=ProblemType.BINARY_CLASSIFICATION,
    model_type='mllib_2.4'
)
pre_prod_asset_deployment = AssetDeploymentRequest(
    deployment_id=pre_prod_deployment_uid,
    name=PRE_PROD_DEPLOYMENT_NAME,
    deployment_type=DeploymentTypes.ONLINE,
    url=pre_prod_deployment["entity"]["status"]["online_url"]["url"]
)

pre_prod_asset_properties_request = AssetPropertiesRequest(
    label_column="Risk",
    probability_fields=["probability"],
    prediction_field="predictedLabel",
    feature_fields=["CheckingStatus","LoanDuration","CreditHistory","LoanPurpose","LoanAmount","ExistingSavings","EmploymentDuration","InstallmentPercent","Sex","OthersOnLoan","CurrentResidenceDuration","OwnsProperty","Age","InstallmentPlans","Housing","ExistingCreditsCount","Job","Dependents","Telephone","ForeignWorker"],
    categorical_fields=["CheckingStatus","CreditHistory","LoanPurpose","ExistingSavings","EmploymentDuration","Sex","OthersOnLoan","OwnsProperty","InstallmentPlans","Housing","Job","Telephone","ForeignWorker"]
)
pre_prod_asset_details
pre_prod_subscription = wos_client.subscriptions.add(
        data_mart_id=data_mart_id,
        service_provider_id=service_provider_id,
        asset=pre_prod_asset,
        deployment=pre_prod_asset_deployment,
        asset_properties=pre_prod_asset_properties_request).result
pre_prod_subscription_id = pre_prod_subscription.metadata.id
print(pre_prod_subscription_id)
# Create challenger subscription
challenger_deployment = wml_client.deployments.get_details(deployment_uid=challenger_deployment_uid)
challenger_asset_details=wos_client.service_providers.get_deployment_asset(data_mart_id=data_mart_id,service_provider_id=service_provider_id,deployment_id=challenger_deployment_uid,deployment_space_id=space_id)
challenger_asset = Asset(
    asset_id=challenger_asset_details["entity"]["asset"]["asset_id"],
    name=challenger_asset_details["entity"]["asset"]["name"],
    url=challenger_asset_details["entity"]["asset"]["url"],
    asset_type=AssetTypes.MODEL,
    input_data_type=InputDataType.STRUCTURED,
    problem_type=ProblemType.BINARY_CLASSIFICATION,
    model_type='scikit-learn_0.23'
)
challenger_asset_deployment = AssetDeploymentRequest(
    deployment_id=challenger_deployment_uid,
    name=PRE_PROD_CHALLENGER_DEPLOYMENT_NAME,
    deployment_type=DeploymentTypes.ONLINE,
    url=challenger_deployment["entity"]["status"]["online_url"]["url"]
)

challenger_asset_properties_request = AssetPropertiesRequest(
    label_column="Risk",
    probability_fields=["probability"],
    prediction_field="prediction",
    feature_fields=["CheckingStatus","LoanDuration","CreditHistory","LoanPurpose","LoanAmount","ExistingSavings","EmploymentDuration","InstallmentPercent","Sex","OthersOnLoan","CurrentResidenceDuration","OwnsProperty","Age","InstallmentPlans","Housing","ExistingCreditsCount","Job","Dependents","Telephone","ForeignWorker"],
    categorical_fields=["CheckingStatus","CreditHistory","LoanPurpose","ExistingSavings","EmploymentDuration","Sex","OthersOnLoan","OwnsProperty","InstallmentPlans","Housing","Job","Telephone","ForeignWorker"]
)
challenger_subscription = wos_client.subscriptions.add(
        data_mart_id=data_mart_id,
        service_provider_id=service_provider_id,
        asset=challenger_asset,
        deployment=challenger_asset_deployment,
        asset_properties=challenger_asset_properties_request).result
challenger_subscription_id = challenger_subscription.metadata.id
print(challenger_subscription_id)
wos_client.subscriptions.show()
pre_prod_subscription_id
challenger_subscription_id<jupyter_output><empty_output><jupyter_text>Patch the training data reference in both the preprod & challenger subscription<jupyter_code>headers = {}
headers["Content-Type"] = "application/json"
headers["Authorization"] = "Bearer {}".format(generate_access_token())

training_data_reference = {
  "connection": {
    "connection_string": "jdbc:db2://dashdb-txn-sbox-yp-dal09-03.services.dal.bluemix.net:50000/BLUDB:retrieveMessagesFromServerOnGetMessage=true;",
    "database_name": "BLUDB",
    "hostname": "dashdb-txn-sbox-yp-dal09-03.services.dal.bluemix.net",
    "password": "<PASSWORD>",
    "username": "cmb91569"
  },
  "location": {
    "schema_name": "CMB91569",
    "table_name": "CREDIT_RISK_TRAIN_DATA"
  },
  "name": "German credit risk training data",
  "type": "db2"
}

payload = [
 {
   "op": "replace",
   "path": "/asset_properties/training_data_reference",
   "value": training_data_reference
 }
]
response = wos_client.subscriptions.update(pre_prod_subscription_id, payload)
response.result.to_dict()
response = wos_client.subscriptions.update(challenger_subscription_id, payload)
response.result.to_dict()<jupyter_output><empty_output><jupyter_text>Score the model so we can configure monitorsNow that the WML service has been bound and the subscription has been created, we need to send a request to the model before we configure OpenScale. This allows OpenScale to create a payload log in the datamart with the correct schema, so it can capture data coming into and out of the model. First, the code gets the model deployment's endpoint URL, and then sends a few records for predictions.<jupyter_code>fields = ["CheckingStatus","LoanDuration","CreditHistory","LoanPurpose","LoanAmount","ExistingSavings","EmploymentDuration","InstallmentPercent","Sex","OthersOnLoan","CurrentResidenceDuration","OwnsProperty","Age","InstallmentPlans","Housing","ExistingCreditsCount","Job","Dependents","Telephone","ForeignWorker"]
values = [
  ["no_checking",13,"credits_paid_to_date","car_new",1343,"100_to_500","1_to_4",2,"female","none",3,"savings_insurance",46,"none","own",2,"skilled",1,"none","yes"],
  ["no_checking",24,"prior_payments_delayed","furniture",4567,"500_to_1000","1_to_4",4,"male","none",4,"savings_insurance",36,"none","free",2,"management_self-employed",1,"none","yes"],
  ["0_to_200",26,"all_credits_paid_back","car_new",863,"less_100","less_1",2,"female","co-applicant",2,"real_estate",38,"none","own",1,"skilled",1,"none","yes"],
  ["0_to_200",14,"no_credits","car_new",2368,"less_100","1_to_4",3,"female","none",3,"real_estate",29,"none","own",1,"skilled",1,"none","yes"],
  ["0_to_200",4,"no_credits","car_new",250,"less_100","unemployed",2,"female","none",3,"real_estate",23,"none","rent",1,"management_self-employed",1,"none","yes"],
  ["no_checking",17,"credits_paid_to_date","car_new",832,"100_to_500","1_to_4",2,"male","none",2,"real_estate",42,"none","own",1,"skilled",1,"none","yes"],
  ["no_checking",33,"outstanding_credit","appliances",5696,"unknown","greater_7",4,"male","co-applicant",4,"unknown",54,"none","free",2,"skilled",1,"yes","yes"],
  ["0_to_200",13,"prior_payments_delayed","retraining",1375,"100_to_500","4_to_7",3,"male","none",3,"real_estate",37,"none","own",2,"management_self-employed",1,"none","yes"]
]

payload_scoring = {"fields": fields,"values": values}
payload = {
    wml_client.deployments.ScoringMetaNames.INPUT_DATA: [payload_scoring]
}
scoring_response = wml_client.deployments.score(pre_prod_deployment_uid, payload)

print('Single record Scoring result:', '\n fields:', scoring_response['predictions'][0]['fields'], '\n values: ', scoring_response['predictions'][0]['values'][0])
time.sleep(10)
pre_prod_pl_data_set_id = None
pre_prod_pl_data_set_id = wos_client.data_sets.list(type=DataSetTypes.PAYLOAD_LOGGING, 
                                                target_target_id=pre_prod_subscription_id, 
                                                target_target_type=TargetTypes.SUBSCRIPTION).result.data_sets[0].metadata.id
if pre_prod_pl_data_set_id is None:
    print("Payload data set not found. Please check subscription status.")
else:
    print("Payload data set id: ", pre_prod_pl_data_set_id)
    pl_records_count = wos_client.data_sets.get_records_count(pre_prod_pl_data_set_id)
    print("Number of records in the payload logging table: {}".format(pl_records_count))
    if pl_records_count == 0:
        print("Automatic payload logging did not happen, performing explicit payload logging.")
        wos_client.data_sets.store_records(data_set_id=pre_prod_pl_data_set_id, request_body=[PayloadRecord(
                       scoring_id="mrm_"+str(uuid.uuid4()),
                       request=payload_scoring,
                       response=scoring_response['predictions'][0]
                   )])
        time.sleep(5)
        pl_records_count = wos_client.data_sets.get_records_count(pre_prod_pl_data_set_id)
        print("Number of records in the payload logging table: {}".format(pl_records_count))
payload = {
    wml_client.deployments.ScoringMetaNames.INPUT_DATA: [payload_scoring]
}
scoring_response = wml_client.deployments.score(challenger_deployment_uid, payload)

print('Single record Scoring result:', '\n fields:', scoring_response['predictions'][0]['fields'], '\n values: ', scoring_response['predictions'][0]['values'][0])
from ibm_watson_openscale.supporting_classes import *
time.sleep(10)
challenger_pl_data_set_id = None
challenger_pl_data_set_id = wos_client.data_sets.list(type=DataSetTypes.PAYLOAD_LOGGING, 
                                                target_target_id=challenger_subscription_id, 
                                                target_target_type=TargetTypes.SUBSCRIPTION).result.data_sets[0].metadata.id
if challenger_pl_data_set_id is None:
    print("Payload data set not found. Please check subscription status.")
else:
    print("Payload data set id: ", challenger_pl_data_set_id)
    pl_records_count = wos_client.data_sets.get_records_count(challenger_pl_data_set_id)
    print("Number of records in the payload logging table: {}".format(pl_records_count))
    if pl_records_count == 0:
        print("Automatic payload logging did not happen, performing explicit payload logging.")
        wos_client.data_sets.store_records(data_set_id=challenger_pl_data_set_id, request_body=[PayloadRecord(
                       scoring_id="mrm_"+str(uuid.uuid4()),
                       request=payload_scoring,
                       response=scoring_response['predictions'][0]
                   )])
        time.sleep(5)
        pl_records_count = wos_client.data_sets.get_records_count(challenger_pl_data_set_id)
        print("Number of records in the payload logging table: {}".format(pl_records_count))<jupyter_output><empty_output><jupyter_text>Quality monitoring Enable quality monitoringThe code below waits ten seconds to allow the payload logging table to be set up before it begins enabling monitors. First, it turns on the quality (accuracy) monitor and sets an alert threshold of 80%. OpenScale will show an alert on the dashboard if the model accuracy measurement (area under the curve, in the case of a binary classifier) falls below this threshold.The second paramater supplied, min_records, specifies the minimum number of feedback records OpenScale needs before it calculates a new measurement. The quality monitor runs hourly, but the accuracy reading in the dashboard will not change until an additional 50 feedback records have been added, via the user interface, the Python client, or the supplied feedback endpoint.<jupyter_code>time.sleep(10)
target = Target(
        target_type=TargetTypes.SUBSCRIPTION,
        target_id=pre_prod_subscription_id
)
parameters = {
    "min_feedback_data_size": 100,
    "threshold": 0.8
}
pre_prod_quality_monitor_details = wos_client.monitor_instances.create(
    data_mart_id=data_mart_id,
    background_mode=False,
    monitor_definition_id=wos_client.monitor_definitions.MONITORS.QUALITY.ID,
    target=target,
    parameters=parameters
).result
time.sleep(10)
target = Target(
        target_type=TargetTypes.SUBSCRIPTION,
        target_id=challenger_subscription_id
)
parameters = {
    "min_feedback_data_size": 100,
    "threshold": 0.8
}
challenger_quality_monitor_details = wos_client.monitor_instances.create(
    data_mart_id=data_mart_id,
    background_mode=False,
    monitor_definition_id=wos_client.monitor_definitions.MONITORS.QUALITY.ID,
    target=target,
    parameters=parameters
).result<jupyter_output><empty_output><jupyter_text>Fairness, drift monitoring and explanations  Fairness configurationThe code below configures fairness monitoring for our model. It turns on monitoring for two features, Sex and Age. In each case, we must specify:Which model feature to monitorOne or more majority groups, which are values of that feature that we expect to receive a higher percentage of favorable outcomesOne or more minority groups, which are values of that feature that we expect to receive a higher percentage of unfavorable outcomesThe threshold at which we would like OpenScale to display an alert if the fairness measurement falls below (in this case, 80%)Additionally, we must specify which outcomes from the model are favourable outcomes, and which are unfavourable. We must also provide the number of records OpenScale will use to calculate the fairness score. In this case, OpenScale's fairness monitor will run hourly, but will not calculate a new fairness rating until at least 100 records have been added. Finally, to calculate fairness, OpenScale must perform some calculations on the training data, so we provide the dataframe containing the data.<jupyter_code>target = Target(
    target_type=TargetTypes.SUBSCRIPTION,
    target_id=pre_prod_subscription_id

)
parameters = {
    "features": [
        {"feature": "Sex",
         "majority": ['male'],
         "minority": ['female']
         },
        {"feature": "Age",
         "majority": [[26, 75]],
         "minority": [[18, 25]]
         }
    ],
    "favourable_class": ["No Risk"],
    "unfavourable_class": ["Risk"],
    "min_records": 100
}
thresholds = [{
    "metric_id": "fairness_value",
    "specific_values": [{
            "applies_to": [{
                "key": "feature",
                "type": "tag",
                "value": "Age"
            }],
            "value": 95
        },
        {
            "applies_to": [{
                "key": "feature",
                "type": "tag",
                "value": "Sex"
            }],
            "value": 95
        }
    ],
    "type": "lower_limit",
    "value": 80.0
}]

pre_prod_fairness_monitor_details = wos_client.monitor_instances.create(
    data_mart_id=data_mart_id,
    background_mode=False,
    monitor_definition_id=wos_client.monitor_definitions.MONITORS.FAIRNESS.ID,
    target=target,
    parameters=parameters,
    thresholds=thresholds).result
pre_prod_fairness_monitor_details.metadata.id
target = Target(
    target_type=TargetTypes.SUBSCRIPTION,
    target_id=challenger_subscription_id

)
parameters = {
    "features": [
        {"feature": "Sex",
         "majority": ['male'],
         "minority": ['female']
         },
        {"feature": "Age",
         "majority": [[26, 75]],
         "minority": [[18, 25]]
         }
    ],
    "favourable_class": ["No Risk"],
    "unfavourable_class": ["Risk"],
    "min_records": 100
}
thresholds = [{
    "metric_id": "fairness_value",
    "specific_values": [{
            "applies_to": [{
                "key": "feature",
                "type": "tag",
                "value": "Age"
            }],
            "value": 95
        },
        {
            "applies_to": [{
                "key": "feature",
                "type": "tag",
                "value": "Sex"
            }],
            "value": 95
        }
    ],
    "type": "lower_limit",
    "value": 80.0
}]

challenger_fairness_monitor_details = wos_client.monitor_instances.create(
    data_mart_id=data_mart_id,
    background_mode=False,
    monitor_definition_id=wos_client.monitor_definitions.MONITORS.FAIRNESS.ID,
    target=target,
    parameters=parameters,
    thresholds=thresholds).result
challenger_fairness_monitor_details.metadata.id<jupyter_output><empty_output><jupyter_text>Drift configurationEnable the drift configuration for both the subscription created with a threshold of 10% and minimal sample as 100 records.<jupyter_code>monitor_instances = wos_client.monitor_instances.list().result.monitor_instances
for monitor_instance in monitor_instances:
    monitor_def_id=monitor_instance.entity.monitor_definition_id
    if monitor_def_id == wos_client.monitor_definitions.MONITORS.DRIFT.ID and monitor_instance.entity.target.target_id == pre_prod_subscription_id:
        wos_client.monitor_instances.delete(monitor_instance.metadata.id)
        print('Deleted existing drift monitor instance with id: ', monitor_instance.metadata.id)


target = Target(
    target_type=TargetTypes.SUBSCRIPTION,
    target_id=pre_prod_subscription_id

)
parameters = {
    "min_samples": 100,
    "train_drift_model": True,
    "enable_model_drift": False,
    "enable_data_drift": True
}

pre_prod_drift_monitor_details = wos_client.monitor_instances.create(
    data_mart_id=data_mart_id,
    background_mode=False,
    monitor_definition_id=wos_client.monitor_definitions.MONITORS.DRIFT.ID,
    target=target,
    parameters=parameters
).result

pre_prod_drift_monitor_details.metadata.id
monitor_instances = wos_client.monitor_instances.list().result.monitor_instances
for monitor_instance in monitor_instances:
    monitor_def_id=monitor_instance.entity.monitor_definition_id
    if monitor_def_id == wos_client.monitor_definitions.MONITORS.DRIFT.ID and monitor_instance.entity.target.target_id == challenger_subscription_id:
        wos_client.monitor_instances.delete(monitor_instance.metadata.id)
        print('Deleted existing drift monitor instance with id: ', monitor_instance.metadata.id)


target = Target(
    target_type=TargetTypes.SUBSCRIPTION,
    target_id=challenger_subscription_id

)
parameters = {
    "min_samples": 100,
    "train_drift_model": True,
    "enable_model_drift": False,
    "enable_data_drift": True
}

challenger_drift_monitor_details = wos_client.monitor_instances.create(
    data_mart_id=data_mart_id,
    background_mode=False,
    monitor_definition_id=wos_client.monitor_definitions.MONITORS.DRIFT.ID,
    target=target,
    parameters=parameters
).result

challenger_drift_monitor_details.metadata.id<jupyter_output><empty_output><jupyter_text>Configure ExplainabilityFinally, we provide OpenScale with the training data to enable and configure the explainability features.<jupyter_code>target = Target(
    target_type=TargetTypes.SUBSCRIPTION,
    target_id=pre_prod_subscription_id
)
parameters = {
    "enabled": True
}
pre_prod_explain_monitor_details = wos_client.monitor_instances.create(
    data_mart_id=data_mart_id,
    background_mode=False,
    monitor_definition_id=wos_client.monitor_definitions.MONITORS.EXPLAINABILITY.ID,
    target=target,
    parameters=parameters
).result

pre_prod_explain_monitor_details.metadata.id

target = Target(
    target_type=TargetTypes.SUBSCRIPTION,
    target_id=challenger_subscription_id
)
parameters = {
    "enabled": True
}
challenger_explain_monitor_details = wos_client.monitor_instances.create(
    data_mart_id=data_mart_id,
    background_mode=False,
    monitor_definition_id=wos_client.monitor_definitions.MONITORS.EXPLAINABILITY.ID,
    target=target,
    parameters=parameters
).result

challenger_explain_monitor_details.metadata.id<jupyter_output><empty_output><jupyter_text>Enable model risk management (MRM) We enable the MRM configuration for both the subscriptions<jupyter_code>target = Target(
    target_type=TargetTypes.SUBSCRIPTION,
    target_id=pre_prod_subscription_id
)
parameters = {
}
pre_prod_mrm_monitor_details = wos_client.monitor_instances.create(
    data_mart_id=data_mart_id,
    background_mode=False,
    monitor_definition_id='mrm',
    target=target,
    parameters=parameters
).result

pre_prod_mrm_instance_id = pre_prod_mrm_monitor_details.metadata.id
target = Target(
    target_type=TargetTypes.SUBSCRIPTION,
    target_id=challenger_subscription_id
)
parameters = {
}
challenger_mrm_monitor_details = wos_client.monitor_instances.create(
    data_mart_id=data_mart_id,
    background_mode=False,
    monitor_definition_id='mrm',
    target=target,
    parameters=parameters
).result

challenger_mrm_instance_id = challenger_mrm_monitor_details.metadata.id
wos_client.monitor_instances.show()<jupyter_output><empty_output><jupyter_text>Patch the integration reference in the pre-prod subscription<jupyter_code>payload = [
  {
    "op": "add",
    "path": "/integration_reference",
    "value": {
        "integrated_system_id": integrated_system_id,
        "external_id": openpages_model_id
    }
  }
]
response = wos_client.subscriptions.update(pre_prod_subscription_id, payload)
response.result.to_dict()
response = wos_client.subscriptions.update(challenger_subscription_id, payload)
response.result.to_dict()<jupyter_output><empty_output><jupyter_text>Create test data sets from the training data<jupyter_code>test_data_1 = pd_data[1:201]
test_data_1.to_csv("german_credit_risk_test_data_1.csv", encoding="utf-8", index=False)
test_data_2 = pd_data[201:401]
test_data_2.to_csv("german_credit_risk_test_data_2.csv", encoding="utf-8", index=False)
test_data_3 = pd_data[401:601]
test_data_3.to_csv("german_credit_risk_test_data_3.csv", encoding="utf-8", index=False)
test_data_4 = pd_data[601:801]
test_data_4.to_csv("german_credit_risk_test_data_4.csv", encoding="utf-8", index=False)<jupyter_output><empty_output><jupyter_text>Function to upload, evaluate and check the status of the evaluationThis function will upload the test data CSV and trigger the risk evaluation. It will iterate and check the status of the evaluation until its finished with a finite wait duration<jupyter_code>def upload_and_evaluate(file_name, mrm_instance_id):
    
    print("Running upload and evaluate for {}".format(file_name))
    import json
    import time
    from datetime import datetime

    status = None
    monitoring_run_id = None
    GET_UPLOAD_AND_EVALUATION_STATUS_RETRIES = 32
    GET_UPLOAD_AND_EVALUATION_STATUS_INTERVAL = 10
    
    if file_name is not None:
        
        headers = {}
        headers["Content-Type"] = "text/csv"
        headers["Authorization"] = "Bearer {}".format(generate_access_token())
        
        POST_EVALUATIONS_URL = WOS_CREDENTIALS["url"] + "/openscale/{0}/v2/monitoring_services/mrm/monitor_instances/{1}/risk_evaluations?test_data_set_name={2}".format(WOS_GUID, mrm_instance_id, file_name)

        with open(file_name) as file:
            f = file.read()
            b = bytearray(f, 'utf-8')

        response = requests.post(POST_EVALUATIONS_URL, data=bytes(b), headers=headers, verify=False)
        if response.ok is False:
            print("Upload and evalaute for {0} failed with {1}: {2}".format(file_name, response.status_code, response.reason))
            return
        
        headers = {}
        headers["Content-Type"] = "application/json"
        headers["Authorization"] = "Bearer {}".format(generate_access_token())

        GET_EVALUATIONS_URL = WOS_CREDENTIALS["url"] + "/openscale/{0}/v2/monitoring_services/mrm/monitor_instances/{1}/risk_evaluations".format(WOS_GUID, mrm_instance_id)
        
        for i in range(GET_UPLOAD_AND_EVALUATION_STATUS_RETRIES):
        
            response = requests.get(GET_EVALUATIONS_URL, headers=headers, verify=False)
            if response.ok is False:
                print("Getting status of upload and evalaute for {0} failed with {1}: {2}".format(file_name, response.status_code, response.reason))
                return

            response = json.loads(response.text)
            if "metadata" in response and "id" in response["metadata"]:
                monitoring_run_id = response["metadata"]["id"]
            if "entity" in response and "status" in response["entity"]:
                status = response["entity"]["status"]["state"]
            
            if status is not None:
                print(datetime.utcnow().strftime('%H:%M:%S'), status.lower())
                if status.lower() in ["finished", "completed"]:
                    break
                elif "error" in status.lower():
                    print(response)
                    break

            time.sleep(GET_UPLOAD_AND_EVALUATION_STATUS_INTERVAL)

    return status, monitoring_run_id<jupyter_output><empty_output><jupyter_text>Perform Risk EvaluationsWe now start performing evaluations of smaller data sets against both the PreProd and Challenger subscriptions<jupyter_code>upload_and_evaluate("german_credit_risk_test_data_1.csv", pre_prod_mrm_instance_id)
upload_and_evaluate("german_credit_risk_test_data_2.csv", pre_prod_mrm_instance_id)
upload_and_evaluate("german_credit_risk_test_data_3.csv", pre_prod_mrm_instance_id)
upload_and_evaluate("german_credit_risk_test_data_4.csv", pre_prod_mrm_instance_id)
upload_and_evaluate("german_credit_risk_test_data_1.csv", challenger_mrm_instance_id)
upload_and_evaluate("german_credit_risk_test_data_2.csv", challenger_mrm_instance_id)
upload_and_evaluate("german_credit_risk_test_data_3.csv", challenger_mrm_instance_id)
upload_and_evaluate("german_credit_risk_test_data_4.csv", challenger_mrm_instance_id)<jupyter_output><empty_output><jupyter_text>Explore the Model Risk Management UIHere is a quick recap of what we have done so far.1. We've deployed two Credit Risk Model to a WML instance that is designated as Pre-Production2. We've created subscriptions of these two model deployments in OpenScale3. Configured all monitors supported by OpenScale for these subscriptions4. We've performed a few risk evaluations against both these susbscription with the same set of test dataNow, please explore the Model Risk Management UI to visualize the results, compare the performance of models, download the evaluation report as PDF. For more information, refer to the Beta Guide section "Work in Watson OpenScale."Link to OpenScale : https://CLUSTER_URL/aiopenscale/insights  Promote pre-production model to production After you have reviewed the evaluation results of the PreProd Vs Challenger and if you make the decision to promote the Challenger model to Production, the first thing you need to do is to deploy the model into a WML instance that is designated as Production instance  Deploy model to production WML instance<jupyter_code>PROD_MODEL_NAME="German Credit Risk Model - Prod"
PROD_DEPLOYMENT_NAME="German Credit Risk Model - Prod"

PROD_SPACE_NAME="prod"
wml_client.spaces.list()

# Find and set the default space
space_name=PROD_SPACE_NAME
spaces = wml_client.spaces.get_details()['resources']
space_id = None
for space in spaces:
    if space['entity']['name'] == space_name:
        space_id = space["metadata"]["id"]
if space_id is None:
    space_id = wml_client.spaces.store(
        meta_props={wml_client.spaces.ConfigurationMetaNames.NAME: space_name})["metadata"]["id"]
wml_client.set.default_space(space_id)
import numpy 
numpy.version.version

import pandas as pd
import json

from pyspark import SparkContext, SQLContext
from pyspark.ml import Pipeline
from pyspark.ml.classification import RandomForestClassifier,GBTClassifier
from pyspark.ml.evaluation import BinaryClassificationEvaluator
from pyspark.ml.feature import StringIndexer, VectorAssembler, IndexToString
from pyspark.sql.types import StructType, DoubleType, StringType, ArrayType

from pyspark.sql import SparkSession
from pyspark import SparkFiles

spark = SparkSession.builder.getOrCreate()
pd_data = pd.read_csv("german_credit_data_biased_training.csv", sep=",", header=0)
spark_df = spark.read.csv(path="german_credit_data_biased_training.csv", sep=",", header=True, inferSchema=True)
spark_df.head()

(train_data, test_data) = spark_df.randomSplit([0.9, 0.1], 24)
print("Number of records for training: " + str(train_data.count()))
print("Number of records for evaluation: " + str(test_data.count()))

si_CheckingStatus = StringIndexer(inputCol='CheckingStatus', outputCol='CheckingStatus_IX')
si_CreditHistory = StringIndexer(inputCol='CreditHistory', outputCol='CreditHistory_IX')
si_LoanPurpose = StringIndexer(inputCol='LoanPurpose', outputCol='LoanPurpose_IX')
si_ExistingSavings = StringIndexer(inputCol='ExistingSavings', outputCol='ExistingSavings_IX')
si_EmploymentDuration = StringIndexer(inputCol='EmploymentDuration', outputCol='EmploymentDuration_IX')
si_Sex = StringIndexer(inputCol='Sex', outputCol='Sex_IX')
si_OthersOnLoan = StringIndexer(inputCol='OthersOnLoan', outputCol='OthersOnLoan_IX')
si_OwnsProperty = StringIndexer(inputCol='OwnsProperty', outputCol='OwnsProperty_IX')
si_InstallmentPlans = StringIndexer(inputCol='InstallmentPlans', outputCol='InstallmentPlans_IX')
si_Housing = StringIndexer(inputCol='Housing', outputCol='Housing_IX')
si_Job = StringIndexer(inputCol='Job', outputCol='Job_IX')
si_Telephone = StringIndexer(inputCol='Telephone', outputCol='Telephone_IX')
si_ForeignWorker = StringIndexer(inputCol='ForeignWorker', outputCol='ForeignWorker_IX')
si_Label = StringIndexer(inputCol="Risk", outputCol="label").fit(spark_df)
label_converter = IndexToString(inputCol="prediction", outputCol="predictedLabel", labels=si_Label.labels)

va_features = VectorAssembler(
inputCols=["CheckingStatus_IX", "CreditHistory_IX", "LoanPurpose_IX", "ExistingSavings_IX",
           "EmploymentDuration_IX", "Sex_IX", "OthersOnLoan_IX", "OwnsProperty_IX", "InstallmentPlans_IX",
           "Housing_IX", "Job_IX", "Telephone_IX", "ForeignWorker_IX", "LoanDuration", "LoanAmount",
           "InstallmentPercent", "CurrentResidenceDuration", "LoanDuration", "Age", "ExistingCreditsCount",
           "Dependents"], outputCol="features")

classifier=GBTClassifier(featuresCol="features")

pipeline = Pipeline(
stages=[si_CheckingStatus, si_CreditHistory, si_EmploymentDuration, si_ExistingSavings, si_ForeignWorker,
        si_Housing, si_InstallmentPlans, si_Job, si_LoanPurpose, si_OthersOnLoan,
        si_OwnsProperty, si_Sex, si_Telephone, si_Label, va_features, classifier, label_converter])

model = pipeline.fit(train_data)
predictions = model.transform(test_data)
evaluator = BinaryClassificationEvaluator(rawPredictionCol="prediction")
auc = evaluator.evaluate(predictions)

print("Accuracy = %g" % auc)

# Remove existing model and deployment
MODEL_NAME=PROD_MODEL_NAME
DEPLOYMENT_NAME=PROD_DEPLOYMENT_NAME

deployment_details = wml_client.deployments.get_details()
for deployment in deployment_details['resources']:
    deployment_id = wml_client.deployments.get_id(deployment)
    model_id = deployment['entity']['asset']['id']
    if deployment['entity']['name'] == DEPLOYMENT_NAME:
        print('Deleting deployment id', deployment_id)
        wml_client.deployments.delete(deployment_id)
        print('Deleting model id', model_id)
        wml_client.repository.delete(model_id)
wml_client.repository.list_models()

# Save Model
software_spec_uid = wml_client.software_specifications.get_id_by_name("spark-mllib_2.4")
model_props_rf = {
    wml_client.repository.ModelMetaNames.NAME: MODEL_NAME,
    wml_client.repository.ModelMetaNames.DESCRIPTION: MODEL_NAME,
    #wml_client.repository.ModelMetaNames.SPACE_UID: space_id,
    wml_client.repository.ModelMetaNames.SOFTWARE_SPEC_UID: software_spec_uid,
    wml_client.repository.ModelMetaNames.TYPE: 'mllib_2.4'
}
published_model_details = wml_client.repository.store_model(model=model, meta_props=model_props_rf, training_data=train_data, pipeline=pipeline)
print(published_model_details)

# List models in the repository
wml_client.repository.list_models()

# Get the model UID
prod_model_uid = wml_client.repository.get_model_uid(published_model_details)
prod_model_uid


# Deploy model
wml_deployments = wml_client.deployments.get_details()
prod_deployment_uid = None
for deployment in wml_deployments['resources']:
    if DEPLOYMENT_NAME == deployment['entity']['name']:
        prod_deployment_uid = wml_client.deployments.get_id(deployment)
        break

if prod_deployment_uid is None:
    print("Deploying model...")
    meta_props = {
        wml_client.deployments.ConfigurationMetaNames.NAME: DEPLOYMENT_NAME,
        wml_client.deployments.ConfigurationMetaNames.DESCRIPTION: DEPLOYMENT_NAME,
        wml_client.deployments.ConfigurationMetaNames.ONLINE: {}
    }
    deployment = wml_client.deployments.create(artifact_uid=prod_model_uid, name=DEPLOYMENT_NAME, meta_props=meta_props)
    prod_deployment_uid = wml_client.deployments.get_id(deployment)

print("Model id: {}".format(prod_model_uid))
print("Deployment id: {}".format(prod_deployment_uid))

prod_deployment_uid=wml_client.deployments.get_uid(deployment)
prod_deployment_uid

fields = ["CheckingStatus","LoanDuration","CreditHistory","LoanPurpose","LoanAmount","ExistingSavings","EmploymentDuration","InstallmentPercent","Sex","OthersOnLoan","CurrentResidenceDuration","OwnsProperty","Age","InstallmentPlans","Housing","ExistingCreditsCount","Job","Dependents","Telephone","ForeignWorker"]
values = [
  ["no_checking",13,"credits_paid_to_date","car_new",1343,"100_to_500","1_to_4",2,"female","none",3,"savings_insurance",46,"none","own",2,"skilled",1,"none","yes"],
  ["no_checking",24,"prior_payments_delayed","furniture",4567,"500_to_1000","1_to_4",4,"male","none",4,"savings_insurance",36,"none","free",2,"management_self-employed",1,"none","yes"],
  ["0_to_200",26,"all_credits_paid_back","car_new",863,"less_100","less_1",2,"female","co-applicant",2,"real_estate",38,"none","own",1,"skilled",1,"none","yes"],
  ["0_to_200",14,"no_credits","car_new",2368,"less_100","1_to_4",3,"female","none",3,"real_estate",29,"none","own",1,"skilled",1,"none","yes"],
  ["0_to_200",4,"no_credits","car_new",250,"less_100","unemployed",2,"female","none",3,"real_estate",23,"none","rent",1,"management_self-employed",1,"none","yes"],
  ["no_checking",17,"credits_paid_to_date","car_new",832,"100_to_500","1_to_4",2,"male","none",2,"real_estate",42,"none","own",1,"skilled",1,"none","yes"],
  ["no_checking",33,"outstanding_credit","appliances",5696,"unknown","greater_7",4,"male","co-applicant",4,"unknown",54,"none","free",2,"skilled",1,"yes","yes"],
  ["0_to_200",13,"prior_payments_delayed","retraining",1375,"100_to_500","4_to_7",3,"male","none",3,"real_estate",37,"none","own",2,"management_self-employed",1,"none","yes"]
]

payload_scoring = {"fields": fields,"values": values}
payload = {
    wml_client.deployments.ScoringMetaNames.INPUT_DATA: [payload_scoring]
}
scoring_response = wml_client.deployments.score(prod_deployment_uid, payload)

print('Single record scoring result:', '\n fields:', scoring_response['predictions'][0]['fields'], '\n values: ', scoring_response['predictions'][0]['values'][0])
wml_client.deployments.get_details(prod_deployment_uid)<jupyter_output><empty_output><jupyter_text>Bind WML machine learning instance as ProdWatson OpenScale needs to be bound to the Watson Machine Learning instance to capture payload data into and out of the model. If a binding with name "WML Prod" already exists, this code will delete that binding a create a new one.**Note**: Binding with name `WML Prod` is assumed to be only created by this notebook.<jupyter_code>SERVICE_PROVIDER_NAME = 'WML Prod'
SERVICE_PROVIDER_DESCRIPTION = 'Prod service provider created by MRM tutorial'
service_providers = wos_client.service_providers.list().result.service_providers
for service_provider in service_providers:
    service_instance_name = service_provider.entity.name
    if service_instance_name == SERVICE_PROVIDER_NAME:
        service_provider_id = service_provider.metadata.id
        wos_client.service_providers.delete(service_provider_id)
        print("Deleted existing service_provider for WML instance: {}".format(service_provider_id))
added_service_provider_result = wos_client.service_providers.add(
        name=SERVICE_PROVIDER_NAME,
        description=SERVICE_PROVIDER_DESCRIPTION,
        service_type=ServiceTypes.WATSON_MACHINE_LEARNING,
        deployment_space_id = space_id,
        operational_space_id = "production",
        credentials=WMLCredentialsCP4D(
            url=WML_CREDENTIALS["url"],
            username=WML_CREDENTIALS["username"],
            password=<PASSWORD>["password"],
            instance_id=WML_CREDENTIALS["instance_id"]
        ),
        background_mode=False
    ).result
service_provider_id = added_service_provider_result.metadata.id
service_provider_id
added_service_provider_result.to_dict()<jupyter_output><empty_output><jupyter_text>Remove existing prod subscriptionThis code removes previous subscription that matches the name `German Credit Risk Model - Prod` as it is expected this subscription is created only via this notebook.<jupyter_code>subscriptions = wos_client.subscriptions.list().result.subscriptions
for subscription in subscriptions:
    sub_name = subscription.entity.asset.name
    if sub_name == PROD_MODEL_NAME:
        wos_client.subscriptions.delete(subscription.metadata.id)
        print('Deleted existing subscription for', sub_name)<jupyter_output><empty_output><jupyter_text>Import configuration settings from pre-prod modelWith MRM we provide a important feature that lets you copy the configuration settings of your pre-production subscription to the production subscription. To try this out1. Navigate to Model Monitors view in Insights dashboard of OpenScale2. Click on the Add to dashboard3. Select the production model deployment from WML production machine learning provider and click on Configure4. In Selections saved dialog, click on Configure monitors4. In Model Governance, click on Begin and choose the same OpenPages model that was linked to the pre-production model and click on Save5. You will be presented with an dialog about the OpenPages model being associated with another subscription and if you'd like to import settings.6. Click on Import and confirmAll the configuration settings are now copied into the production subscriptionNote: The next set of cells should be executed only after finishing the import settings from the OpenScale dashboard  Score the production model so that we can trigger monitorsNow that the production subscription is configured by copying the configuration, there would be schedules created for each of the monitors to run on a scheduled basis. Quality, Fairness and Mrm will run hourly. Drift will run once in three hours.For this demo purpose, we will trigger the monitors on-demand so that we can see the model summary dashboard without having to wait the entire hour. To do that lets first push some records in the Payload Logging table.<jupyter_code>df = pd_data.sample(n=400)
df = df.drop(['Risk'], axis=1)
fields = df.columns.tolist()
values = df.values.tolist()

payload_scoring = {"fields": fields,"values": values}
payload = {
    wml_client.deployments.ScoringMetaNames.INPUT_DATA: [payload_scoring]
}
scoring_response = wml_client.deployments.score(prod_deployment_uid, payload)

print('Single record scoring result:', '\n fields:', scoring_response['predictions'][0]['fields'], '\n values: ', scoring_response['predictions'][0]['values'][0])
subscriptions = wos_client.subscriptions.list().result.subscriptions
for subscription in subscriptions:
    sub_name = subscription.entity.asset.name
    if sub_name == PROD_MODEL_NAME:
        prod_subscription = subscription
prod_subscription_id = prod_subscription.metadata.id
print(prod_subscription_id)
res = wos_client.subscriptions.get(prod_subscription_id).result.to_dict()
res
time.sleep(10)
prod_pl_data_set_id = None
prod_pl_data_set_id = wos_client.data_sets.list(type=DataSetTypes.PAYLOAD_LOGGING, 
                                                target_target_id=prod_subscription_id, 
                                                target_target_type=TargetTypes.SUBSCRIPTION).result.data_sets[0].metadata.id
if prod_pl_data_set_id is None:
    print("Payload data set not found. Please check subscription status.")
else:
    print("Payload data set id: ", prod_pl_data_set_id)
    pl_records_count = wos_client.data_sets.get_records_count(prod_pl_data_set_id)
    print("Number of records in the payload logging table: {}".format(pl_records_count))
    if pl_records_count == 0:
        print("Automatic payload logging did not happen, performing explicit payload logging.")
        wos_client.data_sets.store_records(data_set_id=prod_pl_data_set_id, request_body=[PayloadRecord(
                       scoring_id='mrm_'+str(uuid.uuid4()),
                       request=payload_scoring,
                       response=scoring_response['predictions'][0]
                   )])
        time.sleep(5)
        pl_records_count = wos_client.data_sets.get_records_count(prod_pl_data_set_id)
        print("Number of records in the payload logging table: {}".format(pl_records_count))<jupyter_output><empty_output><jupyter_text>Fetch all monitor instances<jupyter_code>monitor_instances = wos_client.monitor_instances.list(target_target_id=prod_subscription_id).result.monitor_instances
for monitor_instance in monitor_instances:
    monitor_def_id=monitor_instance.entity.monitor_definition_id
    if monitor_def_id == wos_client.monitor_definitions.MONITORS.QUALITY.ID:
        quality_monitor_instance_id = monitor_instance.metadata.id
    if monitor_def_id == wos_client.monitor_definitions.MONITORS.DRIFT.ID:
        drift_monitor_instance_id = monitor_instance.metadata.id
    if monitor_def_id == wos_client.monitor_definitions.MONITORS.FAIRNESS.ID:
        fairness_monitor_instance_id = monitor_instance.metadata.id
    if monitor_def_id == 'mrm':
        mrm_monitor_instance_id = monitor_instance.metadata.id
                    
print("Quality monitor instance id - {0}".format(quality_monitor_instance_id))
print("Drift monitor instance id - {0}".format(drift_monitor_instance_id))
print("Fairness monitor instance id - {0}".format(fairness_monitor_instance_id))
print("MRM monitor instance id - {0}".format(mrm_monitor_instance_id))<jupyter_output><empty_output><jupyter_text>Run on-demand Quality  Feedback loggingThe code below downloads and stores enough feedback data to meet the minimum threshold so that OpenScale can calculate a new accuracy measurement. It then kicks off the accuracy monitor. The monitors run hourly, or can be initiated via the Python API, the REST API, or the graphical user interface.<jupyter_code>feedback_dataset_id = None
feedback_dataset = wos_client.data_sets.list(type=DataSetTypes.FEEDBACK, 
                                                target_target_id=prod_subscription_id, 
                                                target_target_type=TargetTypes.SUBSCRIPTION).result
feedback_dataset_id = feedback_dataset.data_sets[0].metadata.id
if feedback_dataset_id is None:
    print("Feedback data set not found. Please check quality monitor status.")
print(feedback_dataset_id)
!rm additional_feedback_data_v2.json
!wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/additional_feedback_data_v2.json
with open('additional_feedback_data_v2.json') as feedback_file:
    additional_feedback_data = json.load(feedback_file)
wos_client.data_sets.store_records(feedback_dataset_id, request_body=additional_feedback_data, background_mode=False).result
time.sleep(5)
feedback_records_count = wos_client.data_sets.get_records_count(data_set_id=feedback_dataset_id)
print('Number of records in the feedback table: ', feedback_records_count)
if quality_monitor_instance_id is not None:
    run_details = wos_client.monitor_instances.run(monitor_instance_id=quality_monitor_instance_id, background_mode=False).result
wos_client.monitor_instances.show_metrics(monitor_instance_id=quality_monitor_instance_id)<jupyter_output><empty_output><jupyter_text>Run on-demand Drift<jupyter_code>if drift_monitor_instance_id is not None:
    run_details = wos_client.monitor_instances.run(monitor_instance_id=drift_monitor_instance_id, background_mode=False).result
wos_client.monitor_instances.show_metrics(monitor_instance_id=drift_monitor_instance_id)<jupyter_output><empty_output><jupyter_text>Run on-demand Fairness<jupyter_code>if fairness_monitor_instance_id is not None:
    run_details = wos_client.monitor_instances.run(monitor_instance_id=fairness_monitor_instance_id, background_mode=False).result
wos_client.monitor_instances.show_metrics(monitor_instance_id=fairness_monitor_instance_id)<jupyter_output><empty_output><jupyter_text>Run on-demand MRM<jupyter_code>if mrm_monitor_instance_id is not None:
    run_details = wos_client.monitor_instances.run(monitor_instance_id=mrm_monitor_instance_id, triggered_by='user', background_mode=False).result
wos_client.monitor_instances.show_metrics(monitor_instance_id=mrm_monitor_instance_id)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11481276078804205, 'fraction_numerical': 0.01299318649976232, 'mean_word_length': 5.919415258108725, 'pattern_counts': {'":': 107, '<': 163, '<?xml version=': 0, '>': 163, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '38', 'dir': 'jupyter-structured-clean-dedup', 'id': '258001', 'license': 'Apache-2.0', 'n_tokens_mistral': 24124, 'n_tokens_neox': 23115, 'n_words': 4444, 'path': 'Cloud Pak for Data/WML/notebooks/model risk management/MRM E2E with Openpages on CP4D.ipynb', 'repo_name': 'datapresso/watson-openscale-samples', 'size': '75753'}