<jupyter_start><jupyter_text>Working with Watson Machine Learning This notebook should be run in a Watson Studio project, using **Default Python 3.8** runtime environment. It requires service credentials for the following Cloud services:  * Watson OpenScale   * Watson Machine Learning   * Cloud Object Storage  If you have a paid Cloud account, you may also provision a **Db2 Warehouse** service to take full advantage of integration with Watson Studio and continuous learning services.The notebook will train, create and deploy a German Credit Risk model, configure OpenScale to monitor that deployment  Contents- [Setup](setup)- [Model building and deployment](model)- [OpenScale configuration](openscale)- [Create Trainig data json](quality)- [Fairness and explanations](fairness)  Setup   Package installation**NOTE** Using scikit-learn 0.20.2 is requirement for Drift detection model training. if you are training drift detection model using notebook, make sure you have scikit-learn version 0.20.2. Your main model can be of any scikit-learn framework version supported by Watson Machine Learning.<jupyter_code>!pip install --upgrade ibm-watson-machine-learning | tail -n 1
!pip install --upgrade ibm-watson-openscale | tail -n 1<jupyter_output><empty_output><jupyter_text>Action: restart the kernel!  Configure credentials - WOS_CREDENTIALS (CP4D)- WML_CREDENTIALS (CP4D)- DB_CREDENTIALS (DB2 on CP4D)- SCHEMA_NAME- WML_SPACE_ID<jupyter_code>WOS_CREDENTIALS = {
    "url": "***",
    "username": "***",
    "password": "***"
}<jupyter_output><empty_output><jupyter_text>WML credentials example with API key<jupyter_code>WML_CREDENTIALS = {
                   "url": WOS_CREDENTIALS["url"],
                   "username": WOS_CREDENTIALS["username"],
                   "password" : WOS_CREDENTIALS["password"],
                   "instance_id": "wml_local",
                   "version" : "4.0" #If your env is CP4D 4.0 then specify "4.0" instead of "3.5"
                  }
DB_CREDENTIALS = {
    "hostname":"***",
    "username":"***",
    "password":"***",
    "database":"***",
    "port":50000, #provide your actual DB2 port number (as integer value)
    "ssl":"***",
    "sslmode":"***",
    "certificate_base64":"***"}<jupyter_output><empty_output><jupyter_text>Action: Specify created schema name below.<jupyter_code>SCHEMA_NAME = 'AIOSFASTPATHICP'<jupyter_output><empty_output><jupyter_text>In next cells, you will need to paste some credentials to Cloud Object Storage. If you haven't worked with COS yet please visit getting started with COS tutorial. You can find `COS_API_KEY_ID` and `COS_RESOURCE_CRN` variables in **_Service Credentials_** in menu of your COS instance. Used COS Service Credentials must be created with _Role_ parameter set as Writer. Later training data file will be loaded to the bucket of your instance and used as training refecence in subsription.  `COS_ENDPOINT` variable can be found in **_Endpoint_** field of the menu.<jupyter_code>COS_API_KEY_ID = "***"
COS_RESOURCE_CRN = "***"
COS_ENDPOINT = "***" # Current list avaiable at https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints
COS_IAM_AUTH_ENDPOINT = "https://iam.ng.bluemix.net/oidc/token" 
BUCKET_NAME = "***" #example: "credit-risk-training-data"<jupyter_output><empty_output><jupyter_text>Run the notebookAt this point, the notebook is ready to run. You can either run the cells one at a time, or click the **Kernel** option above and select **Restart and Run All** to run all the cells.  Model building and deployment  In this section you will learn how to train Scikit-learn model and next deploy it as web-service using Watson Machine Learning service.  Load the training data from github<jupyter_code>!rm german_credit_data_biased_training.csv
!wget https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/Cloud%20Pak%20for%20Data/WML/assets/data/credit_risk/german_credit_data_biased_training.csv    
import numpy as np
import pandas as pd

training_data_file_name = "german_credit_data_biased_training.csv"
data_df = pd.read_csv(training_data_file_name)<jupyter_output><empty_output><jupyter_text>Explore data<jupyter_code>data_df.head()
print('Columns: ', list(data_df.columns))
print('Number of columns: ', len(data_df.columns))<jupyter_output><empty_output><jupyter_text>As you can see, the data contains twenty one fields. `Risk` field is the one you would like to predict using feedback data.<jupyter_code>print('Number of records: ', data_df.Risk.count())
target_count = data_df.groupby('Risk')['Risk'].count()
target_count<jupyter_output><empty_output><jupyter_text>Save training data to Cloud Object Storage<jupyter_code>import ibm_boto3
from ibm_botocore.client import Config, ClientError

cos_client = ibm_boto3.resource("s3",
    ibm_api_key_id=COS_API_KEY_ID,
    ibm_service_instance_id=COS_RESOURCE_CRN,
    #ibm_auth_endpoint=COS_IAM_AUTH_ENDPOINT,
    config=Config(signature_version="oauth"),
    endpoint_url=COS_ENDPOINT
)
with open(training_data_file_name, "rb") as file_data:
    cos_client.Object(BUCKET_NAME, training_data_file_name).upload_fileobj(
        Fileobj=file_data
    )<jupyter_output><empty_output><jupyter_text>Create a modelIn this section you will learn how to:- Prepare data for training a model- Create machine learning pipeline- Train a model<jupyter_code>MODEL_NAME = "Scikit German Risk Model - for training data - zLinux"
DEPLOYMENT_NAME = "Scikit German Risk Model - for training data - zLinux"<jupyter_output><empty_output><jupyter_text>You will start with importing required libraries<jupyter_code>from sklearn.linear_model import SGDClassifier
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
from sklearn.decomposition import TruncatedSVD
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split<jupyter_output><empty_output><jupyter_text>Splitting the data into train and test<jupyter_code>train_data, test_data = train_test_split(data_df, test_size=0.2)<jupyter_output><empty_output><jupyter_text>Preparing the pipeline<jupyter_code>features_idx = np.s_[0:-1]
all_records_idx = np.s_[:]
first_record_idx = np.s_[0]<jupyter_output><empty_output><jupyter_text>In this step you will encode target column labels into numeric values. You can use `inverse_transform` to decode numeric predictions into labels.<jupyter_code>string_fields = [type(fld) is str for fld in train_data.iloc[first_record_idx, features_idx]]
ct = ColumnTransformer([("ohe", OneHotEncoder(), list(np.array(train_data.columns)[features_idx][string_fields]))])
clf_linear = SGDClassifier(loss='log', penalty='l2', max_iter=1000, tol=1e-5)

pipeline_linear = Pipeline([('ct', ct), ('clf_linear', clf_linear)])<jupyter_output><empty_output><jupyter_text>Train a model<jupyter_code>risk_model = pipeline_linear.fit(train_data.drop('Risk', axis=1), train_data.Risk)<jupyter_output><empty_output><jupyter_text>Evaluate the model<jupyter_code>from sklearn.metrics import roc_auc_score

predictions = risk_model.predict(test_data.drop('Risk', axis=1))
indexed_preds = [0 if prediction=='No Risk' else 1 for prediction in predictions]

real_observations = test_data.Risk.replace('Risk', 1)
real_observations = real_observations.replace('No Risk', 0).values

auc = roc_auc_score(real_observations, indexed_preds)
print(auc)<jupyter_output><empty_output><jupyter_text>Publish the model In this section, the notebook uses the supplied Watson Machine Learning credentials to save the model (including the pipeline) to the WML instance. Previous versions of the model are removed so that the notebook can be run again, resetting all data for another demo.<jupyter_code>import json
from ibm_watson_machine_learning import APIClient

wml_client = APIClient(WML_CREDENTIALS)
wml_client.version<jupyter_output><empty_output><jupyter_text>Listing all the available spaces<jupyter_code>wml_client.spaces.list(limit=10)
WML_SPACE_ID='***' # use space id here
wml_client.set.default_space(WML_SPACE_ID)<jupyter_output><empty_output><jupyter_text>Remove existing model and deployment<jupyter_code>import time
deployments_list = wml_client.deployments.get_details()
for deployment in deployments_list["resources"]:
    model_id = deployment["entity"]["asset"]["id"]
    deployment_id = deployment["metadata"]["id"]
    if deployment["metadata"]["name"] == DEPLOYMENT_NAME:
        print("Deleting deployment id", deployment_id)
        wml_client.deployments.delete(deployment_id)
        time.sleep(5)
        print("Deleting model id", model_id)
        wml_client.repository.delete(model_id)
        time.sleep(5)

wml_client.repository.list_models()

training_data_references = [
                {
                    "id": "product line",
                    "type": "s3",
                    "connection": {
                        "access_key_id": COS_API_KEY_ID,
                        "endpoint_url": COS_ENDPOINT,
                        "resource_instance_id":COS_RESOURCE_CRN
                    },
                    "location": {
                        "bucket": BUCKET_NAME,
                        "path": training_data_file_name,
                    }
                }
            ]

software_spec_uid = wml_client.software_specifications.get_id_by_name("default_py3.8")
print("Software Specification ID: {}".format(software_spec_uid))

model_props = {
        wml_client._models.ConfigurationMetaNames.NAME:"{}".format(MODEL_NAME),
        wml_client._models.ConfigurationMetaNames.TYPE: "scikit-learn_0.23",
        wml_client._models.ConfigurationMetaNames.SOFTWARE_SPEC_UID: software_spec_uid,
        wml_client._models.ConfigurationMetaNames.LABEL_FIELD: "Risk",
    }
print("Storing model ...")

published_model_details = wml_client.repository.store_model(model=risk_model, meta_props=model_props, training_data=data_df.drop(["Risk"], axis=1), training_target=data_df.Risk)
model_uid = wml_client.repository.get_model_uid(published_model_details)
print("Done")
print("Model ID: {}".format(model_uid))
model_details = wml_client.repository.get_details(model_uid)
print(json.dumps(model_details, indent=2))<jupyter_output><empty_output><jupyter_text>Deploy the model The next section of the notebook deploys the model as a RESTful web service in Watson Machine Learning. The deployed model will have a scoring URL you can use to send data to the model for predictions.<jupyter_code>print("Deploying model...")
deployment_details = wml_client.deployments.create(
    model_uid, 
    meta_props={
        wml_client.deployments.ConfigurationMetaNames.NAME: "{}".format(DEPLOYMENT_NAME),
        wml_client.deployments.ConfigurationMetaNames.ONLINE: {}
    }
)
scoring_url = wml_client.deployments.get_scoring_href(deployment_details)
deployment_uid=wml_client.deployments.get_uid(deployment_details)

print("Scoring URL:" + scoring_url)
print("Model id: {}".format(model_uid))
print("Deployment id: {}".format(deployment_uid))<jupyter_output><empty_output><jupyter_text>Score the model<jupyter_code>fields = ["CheckingStatus", "LoanDuration", "CreditHistory", "LoanPurpose", "LoanAmount", "ExistingSavings",
                  "EmploymentDuration", "InstallmentPercent", "Sex", "OthersOnLoan", "CurrentResidenceDuration",
                  "OwnsProperty", "Age", "InstallmentPlans", "Housing", "ExistingCreditsCount", "Job", "Dependents",
                  "Telephone", "ForeignWorker"]
values = [
            ["no_checking", 13, "credits_paid_to_date", "car_new", 1343, "100_to_500", "1_to_4", 2, "female", "none", 3,
             "savings_insurance", 46, "none", "own", 2, "skilled", 1, "none", "yes"],
            ["no_checking", 24, "prior_payments_delayed", "furniture", 4567, "500_to_1000", "1_to_4", 4, "male", "none",
             4, "savings_insurance", 36, "none", "free", 2, "management_self-employed", 1, "none", "yes"],
        ]

scoring_payload = {"input_data": [{"fields": fields, "values": values}]}
predictions = wml_client.deployments.score(deployment_uid, scoring_payload)
predictions<jupyter_output><empty_output><jupyter_text>Configure OpenScale  The notebook will now import the necessary libraries and set up a Python OpenScale client.<jupyter_code>from ibm_cloud_sdk_core.authenticators import CloudPakForDataAuthenticator

from ibm_watson_openscale import *
from ibm_watson_openscale.supporting_classes.enums import *
from ibm_watson_openscale.supporting_classes import *


authenticator = CloudPakForDataAuthenticator(
        url=WOS_CREDENTIALS['url'],
        username=WOS_CREDENTIALS['username'],
        password=WOS_CREDENTIALS['password'],
        disable_ssl_verification=True
    )
#Create client for the default instance id 00000000-0000-0000-0000-000000000000
wos_client = APIClient(service_url=WOS_CREDENTIALS['url'], authenticator=authenticator)
wos_client.version<jupyter_output><empty_output><jupyter_text>Create schema and datamart  Set up datamart Watson OpenScale uses a database to store payload logs and calculated metrics. If database credentials were supplied, the datamart will be created there unless there is an existing datamart. If an OpenScale datamart exists, the existing datamart will be used and no data will be overwritten.Prior instances of the German Credit model will be removed from OpenScale monitoring.<jupyter_code>wos_client.data_marts.show()
data_marts = wos_client.data_marts.list().result.data_marts
if len(data_marts) == 0:
    if DB_CREDENTIALS is not None:
        if SCHEMA_NAME is None: 
            print("Please specify the SCHEMA_NAME and rerun the cell")

        print('Setting up external datamart')
        added_data_mart_result = wos_client.data_marts.add(
                background_mode=False,
                name="WOS Data Mart",
                description="Data Mart created by WOS tutorial notebook",
                database_configuration=DatabaseConfigurationRequest(
                  database_type=DatabaseType.POSTGRESQL,
                    credentials=PrimaryStorageCredentialsLong(
                        hostname=DB_CREDENTIALS['hostname'],
                        username=DB_CREDENTIALS['username'],
                        password=DB_CREDENTIALS['password'],
                        db=DB_CREDENTIALS['database'],
                        port=DB_CREDENTIALS['port'],
                        ssl=True,
                        sslmode=DB_CREDENTIALS['sslmode'],
                        certificate_base64=DB_CREDENTIALS['certificate_base64']
                    ),
                    location=LocationSchemaName(
                        schema_name= SCHEMA_NAME
                    )
                )
             ).result
    else:
        print('Setting up internal datamart')
        added_data_mart_result = wos_client.data_marts.add(
                background_mode=False,
                name="WOS Data Mart",
                description="Data Mart created by WOS tutorial notebook", 
                internal_database = True).result
        
    data_mart_id = added_data_mart_result.metadata.id
    
else:
    data_mart_id=data_marts[0].metadata.id
    print('Using existing datamart {}'.format(data_mart_id))<jupyter_output><empty_output><jupyter_text>Remove existing service provider connected with used  WML instance.  Multiple service providers for the same engine instance are avaiable in Watson OpenScale. To avoid multiple service providers of used WML instance in the tutorial notebook the following code deletes existing service provder(s) and then adds new one.<jupyter_code>SERVICE_PROVIDER_NAME = "WML - for training data"
SERVICE_PROVIDER_DESCRIPTION = "Added by tutorial WOS notebook."
service_providers = wos_client.service_providers.list().result.service_providers
for service_provider in service_providers:
    service_instance_name = service_provider.entity.name
    if service_instance_name == SERVICE_PROVIDER_NAME:
        service_provider_id = service_provider.metadata.id
        wos_client.service_providers.delete(service_provider_id)
        print("Deleted existing service_provider for WML instance: {}".format(service_provider_id))<jupyter_output><empty_output><jupyter_text>Add service provider Watson OpenScale needs to be bound to the Watson Machine Learning instance to capture payload data into and out of the model. **Note:** You can bind more than one engine instance if needed by calling `wos_client.service_providers.add` method. Next, you can refer to particular service provider using `service_provider_id`.<jupyter_code>added_service_provider_result = wos_client.service_providers.add(
        name=SERVICE_PROVIDER_NAME,
        description=SERVICE_PROVIDER_DESCRIPTION,
        service_type=ServiceTypes.WATSON_MACHINE_LEARNING,
        deployment_space_id = WML_SPACE_ID,
        operational_space_id = "production",
        credentials=WMLCredentialsCP4D(
            url=WML_CREDENTIALS["url"],
            username=WML_CREDENTIALS["username"],
            password=WML_CREDENTIALS["password"],
            instance_id=None
        ),
        background_mode=False
    ).result
service_provider_id = added_service_provider_result.metadata.id
# Generate training stats
# Setup info for generating training stats for GCR Model
from ibm_watson_openscale.utils.training_stats import TrainingStats
service_configuration_support = {
    "enable_fairness": True,
    "enable_explainability": True,
    "enable_drift": True
}

training_data_info = {
    "class_label": "Risk",
    "feature_columns": ["CheckingStatus", "LoanDuration", "CreditHistory", "LoanPurpose", "LoanAmount", "ExistingSavings", "EmploymentDuration", "InstallmentPercent", "Sex", "OthersOnLoan", "CurrentResidenceDuration", "OwnsProperty", "Age", "InstallmentPlans", "Housing", "ExistingCreditsCount", "Job", "Dependents", "Telephone", "ForeignWorker"],
    "categorical_columns": ["Sex", "OwnsProperty", "Telephone", "ForeignWorker"]
}

fairness_attributes = [{
   "feature": "Sex", 
   "majority": [
       "male"
   ],
   "minority": [
       "female"
   ],
   "threshold": 0.8
}]

model_type = "binary"
parameters = {
    "favourable_class" :  [ "No Risk" ],
    "unfavourable_class": [ "Risk" ]
}
min_records = 100

# Generate Training stats
enable_explainability = service_configuration_support.get('enable_explainability')
enable_fairness = service_configuration_support.get('enable_fairness')
data_df = pd.read_csv('german_credit_data_biased_training.csv')
training_data_stats = None
if enable_explainability or enable_fairness:
    fairness_inputs = None
    if enable_fairness:
        fairness_inputs = {
                "fairness_attributes": fairness_attributes,
                "min_records" : min_records,
                "favourable_class" :  parameters["favourable_class"],
                "unfavourable_class": parameters["unfavourable_class"]
            }

    input_parameters = {
        "label_column": training_data_info["class_label"],
        "feature_columns": training_data_info["feature_columns"],
        "categorical_columns": training_data_info["categorical_columns"],
        "fairness_inputs": fairness_inputs,  
        "problem_type" : model_type  
    }

    training_stats = TrainingStats(data_df,input_parameters, explain=enable_explainability, fairness=enable_fairness, drop_na=True)
    training_data_stats = training_stats.get_training_statistics()
    training_data_stats["notebook_version"] = 5.0 

print(training_data_stats)
#Create subscription from training stats
prediction_column = "prediction"
probability_columns = ['probability']
predicted_target_column = "prediction"
#predicted_target_column = "***"
subscription_details = wos_client.subscriptions.add(data_mart_id,
    service_provider_id,
    asset = None,
    deployment = None,
    training_data_stats=training_data_stats,
    deployment_id = deployment_uid,
    deployment_space_id = WML_SPACE_ID,
    prediction_field = prediction_column,
    #predicted_target_field = predicted_target_column,
    probability_fields = probability_columns,background_mode = False).result

subscription_id = subscription_details.metadata.id
print(subscription_details)
print("Subscription id {}".format(subscription_id))
#Score payload
from IPython.utils import io
with io.capture_output() as captured:
    !wget https://raw.githubusercontent.com/IBM/watson-openscale-samples/main/IBM%20Cloud/WML/assets/data/credit_risk/german_credit_feed.json -O german_credit_feed.json
!ls -lh german_credit_feed.json

import random, time
import uuid
from ibm_watson_openscale.supporting_classes.payload_record import PayloadRecord

payload_data_set_id = None
payload_data_set_id = wos_client.data_sets.list(type=DataSetTypes.PAYLOAD_LOGGING, 
                                                target_target_id=subscription_id, 
                                                target_target_type=TargetTypes.SUBSCRIPTION).result.data_sets[0].metadata.id
if payload_data_set_id is None:
    print("Payload data set not found. Please check subscription status.")
else:
    print("Payload data set id: ", payload_data_set_id)
    

with open('german_credit_feed.json', 'r') as scoring_file:
    scoring_data = json.load(scoring_file)

fields = scoring_data['fields']
values = []
for _ in range(100):
    values.append(random.choice(scoring_data['values']))
payload_scoring = {"input_data": [{"fields": fields, "values": values}]}

scoring_response = wml_client.deployments.score(deployment_uid, payload_scoring)
time.sleep(10)

#wos_client.data_sets.store_records(data_set_id=payload_data_set_id, request_body=[PayloadRecord(
#                   scoring_id=str(uuid.uuid4()),
#                   request=payload_scoring,
#                   response=scoring_response,
#                   response_time=460
#               )],background_mode=False)
#time.sleep(5)
pl_records_count = wos_client.data_sets.get_records_count(payload_data_set_id)
print("Number of records in the payload logging table: {}".format(pl_records_count))


#Create monitors
# It will create Bias and explain monitors based on what is enabled
print("Creating monitor instances...")
response = wos_client.monitor_instances.create(monitor_definition_id = None, 
                        target = None, data_mart_id = data_mart_id, training_data_stats=training_data_stats, 
                        subscription_id=subscription_id,background_mode=False)
print(response)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0961710103534452, 'fraction_numerical': 0.0077650838986076404, 'mean_word_length': 4.478973105134474, 'pattern_counts': {'":': 46, '<': 101, '<?xml version=': 0, '>': 101, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '165596', 'license': 'Apache-2.0', 'n_tokens_mistral': 6992, 'n_tokens_neox': 6680, 'n_words': 1523, 'path': 'Cloud Pak for Data/zLinux/Watson OpenScale and Watson ML Engine-cp4d scikit learn-training.ipynb', 'repo_name': 'datapresso/watson-openscale-samples', 'size': '22408'}