<jupyter_start><jupyter_text>Monitor german credit risk model with Watson OpenScale This notebook should be run in a Watson Studio project, using **Default Python 3.7** runtime environment. It requires service credentials for the following Cloud services:  * Watson Machine Learning V4  * Watson OpenScale  If you have a paid Cloud account, you may also provision a **Databases for PostgreSQL** or **Db2 Warehouse** service to take full advantage of integration with Watson Studio and continuous learning services. If you choose not to provision this paid service, you can use the free internal PostgreSQL storage with OpenScale, but will not be able to configure continuous learning for your model.The notebook will configure OpenScale to monitor that deployment, and inject seven days' worth of historical records and measurements for viewing in the OpenScale Insights dashboard.  Prerequisite In order to execute this notebook you will need a deployed model. You can perform it with following [notebook](German%20credit%20risk%20prediction%20with%20Scikit%20for%20model%20monitoring.ipynb). Then you will need to copy `deployment_uid` to this notebook.  Learning goalsIn this notebook, you will learn how to:- Retrieve existing deployment in Watson OpenScale-  Make a WatsonOpenscale subscription-  Prepare and run model monitoring and feedback logging-  Run fairness monitor-  Run drift monitor-  Run custom monitors and metrics Contents- [Setup](setup)- [OpenScale configuration](openscale)- [Quality monitor and feedback logging](quality)- [Fairness monitoring and explanations](fairness)- [Drift monitoring](drift)- [Custom monitors and metrics](custom)- [Summary](summary)  1. Set up the environmentBefore you use the sample code in this notebook, you must perform the following setup tasks:-  Create a Watson Machine Learning (WML) Service instance (a free plan is offered and information about how to create the instance can be found here).  Connection to WMLAuthenticate the Watson Machine Learning service on IBM Cloud. You need to provide platform `api_key` and instance `location`.You can use [IBM Cloud CLI](https://cloud.ibm.com/docs/cli/index.html) to retrieve platform API Key and instance location.API Key can be generated in the following way:```ibmcloud loginibmcloud iam api-key-create API_KEY_NAME```In result, get the value of `api_key` from the output.Location of your WML instance can be retrieved in the following way:```ibmcloud login --apikey API_KEY -a https://cloud.ibm.comibmcloud resource service-instance WML_INSTANCE_NAME```In result, get the value of `location` from the output. **Tip**: Your `Cloud API key` can be generated by going to the [**Users** section of the Cloud console](https://cloud.ibm.com/iam/users). From that page, click your name, scroll down to the **API Keys** section, and click **Create an IBM Cloud API key**. Give your key a name and click **Create**, then copy the created key and paste it below. You can also get a service specific url by going to the [**Endpoint URLs** section of the Watson Machine Learning docs](https://cloud.ibm.com/apidocs/machine-learning).  You can check your instance location in your  Watson Machine Learning (WML) Service instance details.You can also get service specific apikey by going to the [**Service IDs** section of the Cloud Console](https://cloud.ibm.com/iam/serviceids).  From that page, click **Create**, then copy the created key and paste it below.**Action**: Enter your `api_key` and `location` in the following cell.<jupyter_code>apikey = 'PASTE YOUR PLATFORM API KEY HERE'
location = 'PASTE YOUR INSTANCE LOCATION HERE'
wml_credentials = {
    "apikey": apikey,
    "url": 'https://' + location + '.ml.cloud.ibm.com'
}<jupyter_output><empty_output><jupyter_text>Install and import the `ibm-watson-machine-learning` package**Note:** `ibm-watson-machine-learning` documentation can be found here.<jupyter_code>!pip install -U ibm-watson-machine-learning
from ibm_watson_machine_learning import APIClient

client = APIClient(wml_credentials)<jupyter_output><empty_output><jupyter_text>Working with spacesFirst of all, you need to create a space that will be used for your work. If you do not have space already created, you can use [Deployment Spaces Dashboard](https://dataplatform.cloud.ibm.com/ml-runtime/spaces?context=cpdaas) to create one.- Click New Deployment Space- Create an empty space- Select Cloud Object Storage- Select Watson Machine Learning instance and press Create- Copy `space_id` and paste it below**Tip**: You can also use SDK to prepare the space for your work. More information can be found [here](https://github.com/IBM/watson-machine-learning-samples/blob/master/cloud/notebooks/python_sdk/instance-management/Space%20management.ipynb).**Action**: Assign space ID below<jupyter_code>space_id = 'PASTE YOUR SPACE ID HERE'<jupyter_output><empty_output><jupyter_text>You can use `list` method to print all existing spaces.<jupyter_code>client.spaces.list(limit=10)<jupyter_output><empty_output><jupyter_text>To be able to interact with all resources available in Watson Machine Learning, you need to set **space** which you will be using.<jupyter_code>client.set.default_space(space_id)<jupyter_output><empty_output><jupyter_text>List deployments.<jupyter_code>client.deployments.list()<jupyter_output><empty_output><jupyter_text>Put `deployment_id` of model deployed in [prerequisite](prereq) notebook in the next cell.<jupyter_code>deployment_uid = "PUT_YOUR_DEPLOYMENT_ID_HERE"<jupyter_output><empty_output><jupyter_text>Get `deployment_url` and `model_uid` from deployment details.<jupyter_code>deployment_url = client.deployments.get_details(deployment_uid)['entity']['status']['online_url']['url']
model_uid = client.deployments.get_details(deployment_uid)['entity']['asset']['id']<jupyter_output><empty_output><jupyter_text>Install and import the `ibm-watson-openscale` package**Note:** `ibm-watson-openscale` documentation can be found here.<jupyter_code>!pip install -U ibm-watson-openscale
!pip install -U ibm-cloud-sdk-core<jupyter_output><empty_output><jupyter_text>This tutorial can use Databases for PostgreSQL, Db2 Warehouse, or a free internal verison of PostgreSQL to create a datamart for OpenScale.If you have previously configured OpenScale, it will use your existing datamart, and not interfere with any models you are currently monitoring. Do not update the cell below.If you do not have a paid Cloud account or would prefer not to provision this paid service, you may use the free internal PostgreSQL service with OpenScale. Do not update the cell below.To provision a new instance of Db2 Warehouse, locate [Db2 Warehouse in the Cloud catalog](https://cloud.ibm.com/catalog/services/db2-warehouse), give your service a name, and click **Create**. Once your instance is created, click the **Service Credentials** link on the left side of the screen. Click the **New credential** button, give your credentials a name, and click **Add**. Your new credentials can be accessed by clicking the **View credentials** button. Copy and paste your Db2 Warehouse credentials into the cell below.To provision a new instance of Databases for PostgreSQL, locate [Databases for PostgreSQL in the Cloud catalog](https://cloud.ibm.com/catalog/services/databases-for-postgresql), give your service a name, and click **Create**. Once your instance is created, click the **Service Credentials** link on the left side of the screen. Click the **New credential** button, give your credentials a name, and click **Add**. Your new credentials can be accessed by clicking the **View credentials** button. Copy and paste your Databases for PostgreSQL credentials into the cell below.<jupyter_code>DB_CREDENTIALS = None
SCHEMA_NAME = None<jupyter_output><empty_output><jupyter_text>__If you previously configured OpenScale to use the free internal version of PostgreSQL, you can switch to a new datamart using a paid database service.__ If you would like to delete the internal PostgreSQL configuration and create a new one using service credentials supplied in the cell above, set the __KEEP_MY_INTERNAL_POSTGRES__ variable below to __False__ below. In this case, the notebook will remove your existing internal PostgreSQL datamart and create a new one with the supplied credentials. __*NO DATA MIGRATION WILL OCCUR.*__<jupyter_code>KEEP_MY_INTERNAL_POSTGRES = True<jupyter_output><empty_output><jupyter_text>In next cells, you will need to paste some credentials to Cloud Object Storage. If you haven't worked with COS yet please visit [getting started with COS tutorial](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-getting-started). You can find `COS_API_KEY_ID` and `COS_RESOURCE_CRN` variables in **_Service Credentials_** in menu of your COS instance. Used COS Service Credentials must be created with _Role_ parameter set as Writer. Later training data file will be loaded to the bucket of your instance and used as training refecence in subsription.  `COS_ENDPOINT` variable can be found in **_Endpoint_** field of the menu.<jupyter_code>COS_API_KEY_ID = "***"
COS_RESOURCE_CRN = "***" 
COS_ENDPOINT = "***" 
IAM_AUTH_ENDPOINT = "***"
BUCKET_NAME = "***"<jupyter_output><empty_output><jupyter_text>Run the notebookAt this point, the notebook is ready to run. You can either run the cells one at a time, or click the **Kernel** option above and select **Restart and Run All** to run all the cells.  Load the training data from github<jupyter_code>!rm german_credit_data_biased_training.csv
!wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/german_credit_data_biased_training.csv<jupyter_output><empty_output><jupyter_text>Retrieve data filename, model name and deployment name.<jupyter_code>training_data_file_name = "german_credit_data_biased_training.csv"
MODEL_NAME = "Scikit German Risk Model WML V4"

DEPLOYMENT_NAME = "Scikit German Risk Deployment WML V4"<jupyter_output><empty_output><jupyter_text>Configure OpenScale  The notebook will now import the necessary libraries and set up a Python OpenScale client.<jupyter_code>from ibm_cloud_sdk_core.authenticators import IAMAuthenticator

from ibm_watson_openscale import APIClient
from ibm_watson_openscale.supporting_classes.enums import *
from ibm_watson_openscale.supporting_classes import *


authenticator = IAMAuthenticator(apikey=wml_credentials["apikey"])
wos_client = APIClient(authenticator=authenticator)
wos_client.version<jupyter_output><empty_output><jupyter_text>Create schema and datamart  Set up datamart Watson OpenScale uses a database to store payload logs and calculated metrics. If database credentials were **not** supplied above, the notebook will use the free, internal lite database. If database credentials were supplied, the datamart will be created there **unless** there is an existing datamart **and** the **KEEP_MY_INTERNAL_POSTGRES** variable is set to **True**. If an OpenScale datamart exists in Db2 or PostgreSQL, the existing datamart will be used and no data will be overwritten.Prior instances of the German Credit model will be removed from OpenScale monitoring.<jupyter_code>wos_client.data_marts.show()
data_marts = wos_client.data_marts.list().result.data_marts
if len(data_marts) == 0:
    if DB_CREDENTIALS is not None:
        if SCHEMA_NAME is None: 
            print("Please specify the SCHEMA_NAME and rerun the cell")

        print("Setting up external datamart")
        added_data_mart_result = wos_client.data_marts.add(
                background_mode=False,
                name="WOS Data Mart",
                description="Data Mart created by WOS tutorial notebook",
                database_configuration=DatabaseConfigurationRequest(
                  database_type=DatabaseType.POSTGRESQL,
                    credentials=PrimaryStorageCredentialsLong(
                        hostname=DB_CREDENTIALS["connection"]["postgres"]["hosts"][0]["hostname"],
                        username=DB_CREDENTIALS["connection"]["postgres"]["authentication"]["username"],
                        password=DB_CREDENTIALS["connection"]["postgres"]["authentication"]["password"],
                        db=DB_CREDENTIALS["connection"]["postgres"]["database"],
                        port=DB_CREDENTIALS["connection"]["postgres"]["hosts"][0]["port"],
                        ssl=True,
                        sslmode=DB_CREDENTIALS["connection"]["postgres"]["query_options"]["sslmode"],
                        certificate_base64=DB_CREDENTIALS["connection"]["postgres"]["certificate"]["certificate_base64"]
                    ),
                    location=LocationSchemaName(
                        schema_name= SCHEMA_NAME
                    )
                )
             ).result
    else:
        print("Setting up internal datamart")
        added_data_mart_result = wos_client.data_marts.add(
                background_mode=False,
                name="WOS Data Mart",
                description="Data Mart created by WOS tutorial notebook", 
                internal_database = True).result
        
    data_mart_id = added_data_mart_result.metadata.id
    
else:
    data_mart_id=data_marts[0].metadata.id
    print("Using existing datamart {}".format(data_mart_id))<jupyter_output><empty_output><jupyter_text>Remove existing service provider connected with used  WML instance.  Multiple service providers for the same engine instance are avaiable in Watson OpenScale. To avoid multiple service providers of used WML instance in the tutorial notebook the following code deletes existing service provder(s) and then adds new one.<jupyter_code>SERVICE_PROVIDER_NAME = "Watson Machine Learning V2"
SERVICE_PROVIDER_DESCRIPTION = "Added by tutorial WOS notebook"
service_providers = wos_client.service_providers.list().result.service_providers
for service_provider in service_providers:
    service_instance_name = service_provider.entity.name
    if service_instance_name == SERVICE_PROVIDER_NAME:
        service_provider_id = service_provider.metadata.id
        wos_client.service_providers.delete(service_provider_id)
        print("Deleted existing service_provider for WML instance: {}".format(service_provider_id))<jupyter_output><empty_output><jupyter_text>Add service provider Watson OpenScale needs to be bound to the Watson Machine Learning instance to capture payload data into and out of the model. **Note:** You can bind more than one engine instance if needed by calling `wos_client.service_providers.add` method. Next, you can refer to particular service provider using `service_provider_id`.<jupyter_code>added_service_provider_result = wos_client.service_providers.add(
        name=SERVICE_PROVIDER_NAME,
        description=SERVICE_PROVIDER_DESCRIPTION,
        service_type=ServiceTypes.WATSON_MACHINE_LEARNING,
        deployment_space_id = space_id,
        operational_space_id = "production",
        credentials=WMLCredentialsCloud(
            apikey=wml_credentials["apikey"],
            url=wml_credentials["url"],
            instance_id=None
        ),
        background_mode=False
    ).result
service_provider_id = added_service_provider_result.metadata.id
wos_client.service_providers.show()<jupyter_output><empty_output><jupyter_text>Subscriptions  Remove existing credit risk subscriptions<jupyter_code>wos_client.subscriptions.show()<jupyter_output><empty_output><jupyter_text>This code removes previous subscriptions to the German Credit model to refresh the monitors with the new model and new data.<jupyter_code>subscriptions = wos_client.subscriptions.list().result.subscriptions
for subscription in subscriptions:
    sub_model_id = subscription.entity.asset.asset_id
    if sub_model_id == model_uid:
        wos_client.subscriptions.delete(subscription.metadata.id)
        print("Deleted existing subscription for model", model_uid)<jupyter_output><empty_output><jupyter_text>Following cells create the model subscription in OpenScale using the Python client API. Note that we need to provide the model unique identifier, and some information about the model itself.<jupyter_code>asset = Asset(
    asset_id=model_uid,
    url=deployment_url,
    asset_type=AssetTypes.MODEL,
    input_data_type=InputDataType.STRUCTURED,
    problem_type=ProblemType.BINARY_CLASSIFICATION
)
asset_deployment = AssetDeploymentRequest(
    deployment_id=deployment_uid,
    name=DEPLOYMENT_NAME,
    deployment_type=DeploymentTypes.ONLINE,
    url=deployment_url
)
training_data_reference = TrainingDataReference(
    type="cos",
    location=COSTrainingDataReferenceLocation(
        bucket=BUCKET_NAME,
        file_name=training_data_file_name
    ),
    connection=COSTrainingDataReferenceConnection.from_dict(
        {
            "resource_instance_id": COS_RESOURCE_CRN,
            "url": COS_ENDPOINT,
            "api_key": COS_API_KEY_ID,
            "iam_url": "https://iam.bluemix.net/oidc/token"
        }
    )
)
asset_properties_request = AssetPropertiesRequest(
    label_column="Risk",
    probability_fields=["probability"],
    prediction_field="prediction",
    feature_fields=["CheckingStatus","LoanDuration","CreditHistory","LoanPurpose","LoanAmount","ExistingSavings","EmploymentDuration","InstallmentPercent","Sex","OthersOnLoan","CurrentResidenceDuration","OwnsProperty","Age","InstallmentPlans","Housing","ExistingCreditsCount","Job","Dependents","Telephone","ForeignWorker"],
    categorical_fields=["CheckingStatus","CreditHistory","LoanPurpose","ExistingSavings","EmploymentDuration","Sex","OthersOnLoan","OwnsProperty","InstallmentPlans","Housing","Job","Telephone","ForeignWorker"],
    training_data_reference=training_data_reference
)
subscription_details = wos_client.subscriptions.add(
        data_mart_id=data_mart_id,
        service_provider_id=service_provider_id,
        asset=asset,
        deployment=asset_deployment,
        asset_properties=asset_properties_request).result
subscription_id = subscription_details.metadata.id
print(subscription_details)
import time

time.sleep(5)
payload_data_set_id = None
payload_data_set_id = wos_client.data_sets.list(type=DataSetTypes.PAYLOAD_LOGGING, 
                                                target_target_id=subscription_id, 
                                                target_target_type=TargetTypes.SUBSCRIPTION).result.data_sets[0].metadata.id
if payload_data_set_id is None:
    print("Payload data set not found. Please check subscription status.")
else:
    print("Payload data set id:", payload_data_set_id)
wos_client.data_sets.show()<jupyter_output><empty_output><jupyter_text>Get subscription list<jupyter_code>wos_client.subscriptions.show()<jupyter_output><empty_output><jupyter_text>Score the model so we can configure monitors Now that the WML service has been bound and the subscription has been created, we need to send a request to the model before we configure OpenScale. This allows OpenScale to create a payload log in the datamart with the correct schema, so it can capture data coming into and out of the model. The sends a few records for predictions.<jupyter_code>fields = ["CheckingStatus","LoanDuration","CreditHistory","LoanPurpose","LoanAmount","ExistingSavings","EmploymentDuration","InstallmentPercent","Sex","OthersOnLoan","CurrentResidenceDuration","OwnsProperty","Age","InstallmentPlans","Housing","ExistingCreditsCount","Job","Dependents","Telephone","ForeignWorker"]
values = [
  ["no_checking",10,"credits_paid_to_date","car_new",1343,"100_to_500","1_to_4",2,"female","none",3,"savings_insurance",46,"none","own",2,"skilled",1,"none","yes"],
  ["no_checking",20,"prior_payments_delayed","furniture",4567,"500_to_1000","1_to_4",4,"male","none",4,"savings_insurance",36,"none","free",2,"management_self-employed",1,"none","yes"],
  ["0_to_200",24,"all_credits_paid_back","car_new",863,"less_100","less_1",2,"female","co-applicant",2,"real_estate",38,"none","own",1,"skilled",1,"none","yes"],
  ["0_to_200",17,"no_credits","car_new",2368,"less_100","1_to_4",3,"female","none",3,"real_estate",29,"none","own",1,"skilled",1,"none","yes"],
  ["0_to_200",66,"no_credits","car_new",250,"less_100","unemployed",2,"female","none",3,"real_estate",23,"none","rent",1,"management_self-employed",1,"none","yes"],
  ["no_checking",14,"credits_paid_to_date","car_new",832,"100_to_500","1_to_4",2,"male","none",2,"real_estate",42,"none","own",1,"skilled",1,"none","yes"],
  ["no_checking",35,"outstanding_credit","appliances",5696,"unknown","greater_7",4,"male","co-applicant",4,"unknown",54,"none","free",2,"skilled",1,"yes","yes"],
  ["0_to_200",23,"prior_payments_delayed","retraining",1375,"100_to_500","4_to_7",3,"male","none",3,"real_estate",37,"none","own",2,"management_self-employed",1,"none","yes"]
]

payload_scoring = {"input_data": [{"fields": fields, "values": values}]}
predictions = client.deployments.score(deployment_uid, payload_scoring)

print("Single record scoring result:", "\n fields:", predictions["predictions"][0]["fields"], "\n values: ", predictions["predictions"][0]["values"][0])
import uuid
from ibm_watson_openscale.supporting_classes.payload_record import PayloadRecord
time.sleep(5)
pl_records_count = wos_client.data_sets.get_records_count(payload_data_set_id)
print("Number of records in the payload logging table: {}".format(pl_records_count))
if pl_records_count == 0:
    print("Payload logging did not happen, performing explicit payload logging.")
    wos_client.data_sets.store_records(data_set_id=payload_data_set_id, request_body=[PayloadRecord(
                   scoring_id=str(uuid.uuid4()),
                   request=payload_scoring,
                   response=predictions,
                   response_time=460
               )])
    time.sleep(5)
    pl_records_count = wos_client.data_sets.get_records_count(payload_data_set_id)
    print("Number of records in the payload logging table: {}".format(pl_records_count))<jupyter_output><empty_output><jupyter_text>Quality monitoring and feedback logging   Enable quality monitoring The code below waits ten seconds to allow the payload logging table to be set up before it begins enabling monitors. First, it turns on the quality (accuracy) monitor and sets an alert threshold of 70%. OpenScale will show an alert on the dashboard if the model accuracy measurement (area under the curve, in the case of a binary classifier) falls below this threshold.The second paramater supplied, min_records, specifies the minimum number of feedback records OpenScale needs before it calculates a new measurement. The quality monitor runs hourly, but the accuracy reading in the dashboard will not change until an additional 50 feedback records have been added, via the user interface, the Python client, or the supplied feedback endpoint.<jupyter_code>import time

time.sleep(10)
target = Target(
        target_type=TargetTypes.SUBSCRIPTION,
        target_id=subscription_id
)
parameters = {
    "min_feedback_data_size": 50
}
quality_monitor_details = wos_client.monitor_instances.create(
    data_mart_id=data_mart_id,
    background_mode=False,
    monitor_definition_id=wos_client.monitor_definitions.MONITORS.QUALITY.ID,
    target=target,
    parameters=parameters
).result
quality_monitor_instance_id = quality_monitor_details.metadata.id<jupyter_output><empty_output><jupyter_text>Feedback logging The code below downloads and stores enough feedback data to meet the minimum threshold so that OpenScale can calculate a new accuracy measurement. It then kicks off the accuracy monitor. The monitors run hourly, or can be initiated via the Python API, the REST API, or the graphical user interface.<jupyter_code>!rm additional_feedback_data_v2.json
!wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/additional_feedback_data_v2.json<jupyter_output><empty_output><jupyter_text>Get feedback logging dataset ID<jupyter_code>feedback_dataset_id = None
feedback_dataset = wos_client.data_sets.list(type=DataSetTypes.FEEDBACK, 
                                                target_target_id=subscription_id, 
                                                target_target_type=TargetTypes.SUBSCRIPTION).result
print(feedback_dataset)
feedback_dataset_id = feedback_dataset.data_sets[0].metadata.id
if feedback_dataset_id is None:
    print("Feedback data set not found. Please check quality monitor status.")
import json
with open('additional_feedback_data_v2.json') as feedback_file:
    additional_feedback_data = json.load(feedback_file)
wos_client.data_sets.store_records(feedback_dataset_id, request_body=additional_feedback_data, background_mode=False)
wos_client.data_sets.get_records_count(data_set_id=feedback_dataset_id)
run_details = wos_client.monitor_instances.run(monitor_instance_id=quality_monitor_instance_id, background_mode=False).result
wos_client.monitor_instances.show_metrics(monitor_instance_id=quality_monitor_instance_id)<jupyter_output><empty_output><jupyter_text>Fairness monitoring and explanations   Fairness configurationThe code below configures fairness monitoring for our model. It turns on monitoring for two features, Sex and Age. In each case, we must specify:  * Which model feature to monitor  * One or more **majority** groups, which are values of that feature that we expect to receive a higher percentage of favorable outcomes  * One or more **minority** groups, which are values of that feature that we expect to receive a higher percentage of unfavorable outcomes  * The threshold at which we would like OpenScale to display an alert if the fairness measurement falls below (in this case, 95%)Additionally, we must specify which outcomes from the model are favourable outcomes, and which are unfavourable. We must also provide the number of records OpenScale will use to calculate the fairness score. In this case, OpenScale's fairness monitor will run hourly, but will not calculate a new fairness rating until at least 200 records have been added. Finally, to calculate fairness, OpenScale must perform some calculations on the training data, so we provide the dataframe containing the data.<jupyter_code>wos_client.monitor_instances.show()
target = Target(
    target_type=TargetTypes.SUBSCRIPTION,
    target_id=subscription_id

)
parameters = {
    "features": [
        {"feature": "Sex",
         "majority": ['male'],
         "minority": ['female'],
         "threshold": 0.95
         },
        {"feature": "Age",
         "majority": [[26, 75]],
         "minority": [[18, 25]],
         "threshold": 0.95
         }
    ],
    "favourable_class": ["No Risk"],
    "unfavourable_class": ["Risk"],
    "min_records": 4
}

fairness_monitor_details = wos_client.monitor_instances.create(
    data_mart_id=data_mart_id,
    background_mode=False,
    monitor_definition_id=wos_client.monitor_definitions.MONITORS.FAIRNESS.ID,
    target=target,
    parameters=parameters).result
fairness_monitor_instance_id =fairness_monitor_details.metadata.id<jupyter_output><empty_output><jupyter_text>Drift configuration<jupyter_code>monitor_instances = wos_client.monitor_instances.list().result.monitor_instances
for monitor_instance in monitor_instances:
    monitor_def_id=monitor_instance.entity.monitor_definition_id
    if monitor_def_id == "drift" and monitor_instance.entity.target.target_id == subscription_id:
        wos_client.monitor_instances.delete(monitor_instance.metadata.id)
        print('Deleted existing drift monitor instance with id: ', monitor_instance.metadata.id)


target = Target(
    target_type=TargetTypes.SUBSCRIPTION,
    target_id=subscription_id

)
parameters = {
    "min_samples": 20,
    "train_drift_model": True,
    "enable_model_drift": False,
    "enable_data_drift": True
}

drift_monitor_details = wos_client.monitor_instances.create(
    data_mart_id=data_mart_id,
    background_mode=False,
    monitor_definition_id=wos_client.monitor_definitions.MONITORS.DRIFT.ID,
    target=target,
    parameters=parameters
).result

drift_monitor_instance_id = drift_monitor_details.metadata.id<jupyter_output><empty_output><jupyter_text>Score the model again now that monitoring is configured This next section randomly selects 200 records from the data feed and sends those records to the model for predictions. This is enough to exceed the minimum threshold for records set in the previous section, which allows OpenScale to begin calculating fairness.<jupyter_code>from IPython.utils import io
!rm german_credit_feed.json
with io.capture_output() as captured:
    !wget https://raw.githubusercontent.com/pmservice/ai-openscale-tutorials/master/assets/historical_data/german_credit_risk/wml/german_credit_feed.json -O german_credit_feed.json
!ls -lh german_credit_feed.json<jupyter_output><empty_output><jupyter_text>Score 200 randomly chosen records<jupyter_code>import random

with open('german_credit_feed.json', 'r') as scoring_file:
    scoring_data = json.load(scoring_file)

fields = scoring_data['fields']
values = []
for _ in range(200):
    values.append(random.choice(scoring_data['values']))
payload_scoring = {"input_data": [{"fields": fields, "values": values}]}

scoring_response = client.deployments.score(deployment_uid, payload_scoring)
time.sleep(5)

if pl_records_count == 8:
    print("Payload logging did not happen, performing explicit payload logging.")
    wos_client.data_sets.store_records(data_set_id=payload_data_set_id, request_body=[PayloadRecord(
                   scoring_id=str(uuid.uuid4()),
                   request=payload_scoring,
                   response=scoring_response,
                   response_time=460
               )])
    time.sleep(5)
    pl_records_count = wos_client.data_sets.get_records_count(payload_data_set_id)
    print("Number of records in the payload logging table: {}".format(pl_records_count))
print("Number of records in payload table:", wos_client.data_sets.get_records_count(data_set_id=payload_data_set_id))<jupyter_output><empty_output><jupyter_text>Run fairness monitor Kick off a fairness monitor run on current data. The monitor runs hourly, but can be manually initiated using the Python client, the REST API, or the graphical user interface.<jupyter_code>run_details = wos_client.monitor_instances.run(monitor_instance_id=fairness_monitor_instance_id, background_mode=False)
wos_client.monitor_instances.show_metrics(monitor_instance_id=fairness_monitor_instance_id)<jupyter_output><empty_output><jupyter_text>Run drift monitorKick off a drift monitor run on current data. The monitor runs every hour, but can be manually initiated using the Python client, the REST API.<jupyter_code>drift_run_details = wos_client.monitor_instances.run(monitor_instance_id=drift_monitor_instance_id, background_mode=False)
wos_client.monitor_instances.show_metrics(monitor_instance_id=drift_monitor_instance_id)<jupyter_output><empty_output><jupyter_text>Configure Explainability Finally, we provide OpenScale with the training data to enable and configure the explainability features.<jupyter_code>target = Target(
    target_type=TargetTypes.SUBSCRIPTION,
    target_id=subscription_id
)
parameters = {
    "enabled": True
}
explainability_details = wos_client.monitor_instances.create(
    data_mart_id=data_mart_id,
    background_mode=False,
    monitor_definition_id=wos_client.monitor_definitions.MONITORS.EXPLAINABILITY.ID,
    target=target,
    parameters=parameters
).result

explainability_monitor_id = explainability_details.metadata.id<jupyter_output><empty_output><jupyter_text>Run explanation for sample record<jupyter_code>pl_records_resp = wos_client.data_sets.get_list_of_records(data_set_id=payload_data_set_id, limit=1, offset=0).result
scoring_ids = [pl_records_resp["records"][0]["entity"]["values"]["scoring_id"]]
print("Running explanations on scoring IDs: {}".format(scoring_ids))
explanation_types = ["lime", "contrastive"]
result = wos_client.monitor_instances.explanation_tasks(scoring_ids=scoring_ids, explanation_types=explanation_types).result
print(result)<jupyter_output><empty_output><jupyter_text>Custom monitors and metrics   Register custom monitor<jupyter_code>def get_definition(monitor_name):
    monitor_definitions = wos_client.monitor_definitions.list().result.monitor_definitions
    
    for definition in monitor_definitions:
        if monitor_name == definition.entity.name:
            return definition
    
    return None
monitor_name = 'my model performance'
metrics = [MonitorMetricRequest(name='sensitivity',
                                thresholds=[MetricThreshold(type=MetricThresholdTypes.LOWER_LIMIT, default=0.8)]),
          MonitorMetricRequest(name='specificity',
                                thresholds=[MetricThreshold(type=MetricThresholdTypes.LOWER_LIMIT, default=0.75)])]
tags = [MonitorTagRequest(name='region', description='customer geographical region')]

existing_definition = get_definition(monitor_name)

if existing_definition is None:
    custom_monitor_details = wos_client.monitor_definitions.add(name=monitor_name, metrics=metrics, tags=tags, background_mode=False).result
else:
    custom_monitor_details = existing_definition<jupyter_output><empty_output><jupyter_text>Show available monitors types<jupyter_code>wos_client.monitor_definitions.show()<jupyter_output><empty_output><jupyter_text>Get monitors uids and details<jupyter_code>custom_monitor_id = custom_monitor_details.metadata.id

print(custom_monitor_id)
custom_monitor_details = wos_client.monitor_definitions.get(monitor_definition_id=custom_monitor_id).result
print('Monitor definition details:', custom_monitor_details)<jupyter_output><empty_output><jupyter_text>Enable custom monitor for subscription<jupyter_code>target = Target(
        target_type=TargetTypes.SUBSCRIPTION,
        target_id=subscription_id
    )

thresholds = [MetricThresholdOverride(metric_id='sensitivity', type = MetricThresholdTypes.LOWER_LIMIT, value=0.9)]

custom_monitor_instance_details = wos_client.monitor_instances.create(
            data_mart_id=data_mart_id,
            background_mode=False,
            monitor_definition_id=custom_monitor_id,
            target=target
).result<jupyter_output><empty_output><jupyter_text>Get monitor instance id and configuration details<jupyter_code>custom_monitor_instance_id = custom_monitor_instance_details.metadata.id
custom_monitor_instance_details = wos_client.monitor_instances.get(custom_monitor_instance_id).result
print(custom_monitor_instance_details)<jupyter_output><empty_output><jupyter_text>Storing custom metrics<jupyter_code>from datetime import datetime, timezone, timedelta
from ibm_watson_openscale.base_classes.watson_open_scale_v2 import MonitorMeasurementRequest
custom_monitoring_run_id = "11122223333111abc"
measurement_request = [MonitorMeasurementRequest(timestamp=datetime.now(timezone.utc), 
                                                 metrics=[{"specificity": 0.78, "sensitivity": 0.67, "region": "us-south"}], run_id=custom_monitoring_run_id)]
print(measurement_request[0])
published_measurement_response = wos_client.monitor_instances.measurements.add(
    monitor_instance_id=custom_monitor_instance_id,
    monitor_measurement_request=measurement_request).result
published_measurement_id = published_measurement_response[0]["measurement_id"]
print(published_measurement_response)<jupyter_output><empty_output><jupyter_text>List and get custom metrics<jupyter_code>time.sleep(5)
published_measurement = wos_client.monitor_instances.measurements.get(monitor_instance_id=custom_monitor_instance_id, measurement_id=published_measurement_id).result
print(published_measurement)
print('Datamart:', data_mart_id)
print('Model:', model_uid)
print('Deployment:', deployment_uid)<jupyter_output><empty_output><jupyter_text>Identify transactions for Explainability Transaction IDs identified by the cells below can be copied and pasted into the Explainability tab of the OpenScale dashboard.<jupyter_code>wos_client.data_sets.show_records(data_set_id=payload_data_set_id, limit=5)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08867240472698087, 'fraction_numerical': 0.008136606426258545, 'mean_word_length': 5.655737704918033, 'pattern_counts': {'":': 33, '<': 169, '<?xml version=': 0, '>': 169, 'https://': 15, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '42', 'dir': 'jupyter-structured-clean-dedup', 'id': '210329', 'license': 'Apache-2.0', 'n_tokens_mistral': 10624, 'n_tokens_neox': 10215, 'n_words': 2710, 'path': 'cloud/notebooks/python_sdk/monitoring/Monitor credit risk model with Watson Openscale.ipynb', 'repo_name': 'kelvinlui/watson-machine-learning-samples', 'size': '36133'}