<jupyter_start><jupyter_text>Best heart drug prediction using Watson Machine Learning                        This notebook contains steps and code to train, deploy, and start scoring new data. This notebook introduces commands for getting data, model persistance to Watson Machine Learning repository, model deployment, and scoring.Some familiarity with Python is helpful. This notebook uses Python 3 and Apache Spark 2.1.You will use the data set published on git, **drug_feedback_data.csv**, which contains anonymous information about patients records. Use the details of this data set to predict the best drug to treat heart disease. Learning goalsThis notebook teaches you how to:-  Publish a sample model in the Watson Machine Learning (WML) repositoryYou will also learn how to use the WML API to:-  Deploy a model for online scoring  ContentsThis notebook contains the following parts:1.	[Set up the environment](setup)2.	[Create spark ml model](model)3.	[Persist model](load)4.	[Deploy & score](score)5.	[Summary and next steps](summary)  1. Set up the environmentBefore you use the sample code in this notebook, you must perform the following setup tasks:-  Create a [Watson Machine Learning (WML) Service](https://console.ng.bluemix.net/catalog/services/ibm-watson-machine-learning/) instance (a free plan is offered and information about how to create the instance is [here](https://dataplatform.ibm.com/docs/content/analyze-data/wml-setup.html))- Create a [Spark Service](https://console.ng.bluemix.net/catalog/services/spark/) instance (an entry plan is offered).- Create a [Db2 Warehouse on Cloud Service](https://console.bluemix.net/catalog/services/db2-warehouse-on-cloud/) instance (an entry plan is offered).- Create the **DRUG_TRAIN_DATA_UPDATED** table in **Db2 Warehouse on Cloud**.   + Download [drug_train_data_updated.csv](https://raw.githubusercontent.com/pmservice/wml-sample-models/master/spark/drug-selection/data/drug_train_data_updated.csv) file from git repository.  + Click **Open the console** to get started with **Db2 Warehouse on Cloud** icon.  + Select the **Load Data** and **Desktop** load type.  + **Drag and drop** previously downloaded file and press **Next**.  + Select **Schema** to import data and click **New Table**.   + Write the name **DRUG_TRAIN_DATA_UPDATED** for **new table** than click **Next** to finish data import.  + Use `;` as **field separator**.  + Click **Next** to create a table with the uploaded data.  2. Create the spark machine learning model In this section you will learn how to prepare data, create an Apache Spark machine learning pipeline, and train a model.- [2.1 Load the training data from Db2 Warehouse on Cloud](load)- [2.2 Prepare the data](prep)- [2.3 Create the pipeline](pipe)- [2.4 Train the model](train)  2.1 Load the training data from Db2 Warehouse on Cloud Run the following cell to the load the DRUG_TRAIN_DATA_UPDATED table content into the Spark DataFrame. Enter your authentication data as required. **Tip:** The authentication information can be found under the **Service Credentials**  tab of Db2 Warehouse on Cloud service instance created in IBM Cloud. Click **New credential** to create credentials if you do not have any.<jupyter_code>db2_service_credentials = {
  "port": 50000,
  "db": "BLUDB",
  "username": "*****",
  "ssljdbcurl": "jdbc:db2://dashdb-entry-yp-dal10-01.services.dal.bluemix.net:50001/BLUDB:sslConnection=true;",
  "host": "dashdb-entry-yp-dal10-01.services.dal.bluemix.net",
  "https_url": "https://dashdb-entry-yp-dal10-01.services.dal.bluemix.net:8443",
  "dsn": "***",
  "hostname": "dashdb-entry-yp-dal10-01.services.dal.bluemix.net",
  "jdbcurl": "***",
  "ssldsn": "***",
  "uri": "***",
  "password": "***"
}
db2_credentials = {
    'driver': 'com.ibm.db2.jcc.DB2Driver',
    'jdbcurl': db2_service_credentials['jdbcurl'],
    'user': db2_service_credentials['username'],
    'password': <PASSWORD>['password']
}
tablename = "{schema}.{table}".format(schema=db2_credentials['user'], table='DRUG_TRAIN_DATA_UPDATED')
DRUG_TRAIN_DATA_UPDATED_data = spark.read.jdbc(db2_credentials['jdbcurl'], table=tablename, properties=db2_credentials)
DRUG_TRAIN_DATA_UPDATED_data.show(5)<jupyter_output><empty_output><jupyter_text>The DRUG column is the target/label column.  2.2 Prepare the data In this subsection you will split your data into two data sets: - Train data set- Test data set<jupyter_code>(train_data, test_data) = DRUG_TRAIN_DATA_UPDATED_data.randomSplit([0.8, 0.2], 24)

print("Number of records for training: " + str(train_data.count()))
print("Number of records for evaluation: " + str(test_data.count()))<jupyter_output><empty_output><jupyter_text>As you can see, your data has been successfully split into two data sets: - The train data set, which is the largest group, is used for training. - The test data set is used for model evaluation.  2.3 Create the pipeline In this section you will create an Apache Spark machine learning pipeline. First, import the Apache Spark machine learning packages that will be needed in the subsequent steps.<jupyter_code>from pyspark.ml.feature import OneHotEncoder, StringIndexer, IndexToString, VectorAssembler
from pyspark.ml.classification import DecisionTreeClassifier
from pyspark.ml.evaluation import MulticlassClassificationEvaluator
from pyspark.ml import Pipeline, Model<jupyter_output><empty_output><jupyter_text>In the following step, use the StringIndexer transformer to convert all the string fields to numeric ones.<jupyter_code>stringIndexer_sex = StringIndexer(inputCol = 'SEX', outputCol = 'SEX_IX')
stringIndexer_bp = StringIndexer(inputCol = 'BP', outputCol = 'BP_IX')
stringIndexer_chol = StringIndexer(inputCol = 'CHOLESTEROL', outputCol = 'CHOL_IX')
stringIndexer_label = StringIndexer(inputCol="DRUG", outputCol="label").fit(DRUG_TRAIN_DATA_UPDATED_data)<jupyter_output><empty_output><jupyter_text>Create a feature vector by combining all the features together.<jupyter_code>vectorAssembler_features = VectorAssembler(inputCols=["AGE", "SEX_IX", "BP_IX", "CHOL_IX", "NA", "K"], outputCol="features")<jupyter_output><empty_output><jupyter_text>Next, define the estimators you want to use for classification. Decision Tree is used in the following example.<jupyter_code>dt = DecisionTreeClassifier(labelCol="label", featuresCol="features")<jupyter_output><empty_output><jupyter_text>Finally, convert the indexed labels back to the original labels.<jupyter_code>labelConverter = IndexToString(inputCol="prediction", outputCol="predictedLabel", labels=stringIndexer_label.labels)<jupyter_output><empty_output><jupyter_text>Build the pipeline. A pipeline consists of transformers and an estimator.<jupyter_code>pipeline_dt = Pipeline(stages=[stringIndexer_label, stringIndexer_sex, stringIndexer_bp, stringIndexer_chol, vectorAssembler_features, dt, labelConverter])<jupyter_output><empty_output><jupyter_text>2.4 Train the model Now, you can train your Decision Tree model by using the previously defined pipeline and train data.<jupyter_code>model = pipeline_dt.fit(train_data)<jupyter_output><empty_output><jupyter_text>You can check your model accuracy now. Use test data to evaluate the model.<jupyter_code>predictions = model.transform(test_data)
evaluatorDT = MulticlassClassificationEvaluator(labelCol="label", predictionCol="prediction", metricName="accuracy")
accuracy = evaluatorDT.evaluate(predictions)

print("Accuracy = %g" % accuracy)<jupyter_output><empty_output><jupyter_text>You can tune your model now to achieve better accuracy. To keep this example simple, the tuning section is omitted.  3. Store the model In this section you will learn how to store sample model in Watson Machine Learning repository by using repository client. First, install and import the client library.<jupyter_code>!rm -rf $PIP_BUILD/watson-machine-learning-client
!pip install watson-machine-learning-client --upgrade<jupyter_output><empty_output><jupyter_text>**Note**: Apache Spark 2.1 is required.<jupyter_code>from watson_machine_learning_client import WatsonMachineLearningAPIClient
import json<jupyter_output><empty_output><jupyter_text>Authenticate to the Watson Machine Learning service on IBM Cloud.**Tip**: Authentication information (your credentials) can be found in the Service Credentials tab of the service instance that you created on IBM Cloud. If you cannot see the **instance_id** field in **Service Credentials**, click **New credential (+)** to generate new authentication information. **Action**: Enter your Watson Machine Learning service instance credentials here.<jupyter_code>wml_credentials={
  "url": "https://us-south.ml.cloud.ibm.com",
  "username": "***",
  "password": "***",
  "instance_id": "***"
}<jupyter_output><empty_output><jupyter_text>Create the WatsonMachineLearningAPIClient.<jupyter_code>client = WatsonMachineLearningAPIClient(wml_credentials)<jupyter_output><empty_output><jupyter_text>Prepare the metadata **Tip**: If the accuracy value falls below the threshold value, retraining action is required. Prepare the additional information to be saved as model's metadata:* TRAINING_DATA_REFERENCE* OUTPUT_DATA_SCHEMA* EVALUATION_METHOD: **multiclass*** EVALUATION_METRICS name: **accuracy** (metric name used to evaluate the model)* EVALUATION_METRICS value: **0.87** (accuracy value calculated few steps above)* EVALUATION_METRICS threshold: **0.8** (if the accuracy after evaluation using feedback data is below this threshold auto-retraining is triggered) **Tip**: All required fields can be found on Service Credentials tab of Db2 Warehouse on Cloud service instance created in IBM Cloud.<jupyter_code>training_data_reference = {
 "name": "DRUG feedback",
 "connection": db2_service_credentials,
 "source": {
  "tablename": "DRUG_TRAIN_DATA_UPDATED",
  "type": "dashdb"
 }
}<jupyter_output><empty_output><jupyter_text>Define OUTPUT_DATA_SCHEMA<jupyter_code>train_data_schema = train_data.schema
label_field = next(f for f in train_data_schema.fields if f.name == "DRUG")
label_field.metadata['values'] = stringIndexer_label.labels<jupyter_output><empty_output><jupyter_text>Set up modelling roles in OUTPUT_DATA_SCHEMA<jupyter_code>from pyspark.sql.types import *

input_fileds = filter(lambda f: f.name != "DRUG", train_data_schema.fields)

output_data_schema = StructType(list(input_fileds)). \
    add("prediction", DoubleType(), True, {'modeling_role': 'prediction'}). \
    add("predictedLabel", StringType(), True, {'modeling_role': 'decoded-target', 'values': stringIndexer_label.labels}). \
    add("probability", ArrayType(DoubleType()), True, {'modeling_role': 'probability'})<jupyter_output><empty_output><jupyter_text>Add all the information to model meta props.<jupyter_code>model_props = {
    client.repository.ModelMetaNames.NAME: "drug-selection",
    client.repository.ModelMetaNames.TRAINING_DATA_REFERENCE: training_data_reference,
    client.repository.ModelMetaNames.OUTPUT_DATA_SCHEMA: output_data_schema.jsonValue(),
    client.repository.ModelMetaNames.EVALUATION_METHOD: "multiclass",
    client.repository.ModelMetaNames.EVALUATION_METRICS: [
        {
           "name": "accuracy",
           "value": accuracy,
           "threshold": 0.8
        }
    ]
}<jupyter_output><empty_output><jupyter_text>Store the model.<jupyter_code>published_model_details = client.repository.store_model(model=model, meta_props=model_props, training_data=train_data, pipeline=pipeline_dt)
model_uid = client.repository.get_model_uid(published_model_details)<jupyter_output><empty_output><jupyter_text>**Tip**: Use `client.repository.ModelMetaNames.show()` to get the list of available props.  4. Deploy and score You can udeploy previously stored model as web service.<jupyter_code>deployment_details = client.deployments.create(model_uid, 'best-drug model deployment')<jupyter_output><empty_output><jupyter_text>In next step, get the scoring endpoint.<jupyter_code>scoring_endpoint = client.deployments.get_scoring_url(deployment_details)
print(scoring_endpoint)
scoring_payload = {
    "fields": ["AGE", "SEX", "BP", "CHOLESTEROL","NA","K"],
    "values": [[20.0, "F", "HIGH", "HIGH", 0.71, 0.07], [55.0, "M", "LOW", "HIGH", 0.71, 0.07]]
}<jupyter_output><empty_output><jupyter_text>Score the model using sample scoring records and scoring_enpoint.<jupyter_code>score = client.deployments.score(scoring_endpoint, scoring_payload)

print(str(score))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10856269113149847, 'fraction_numerical': 0.008932882665379043, 'mean_word_length': 6.4951749095295535, 'pattern_counts': {'":': 26, '<': 90, '<?xml version=': 0, '>': 90, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '392697', 'license': 'Apache-2.0', 'n_tokens_mistral': 3835, 'n_tokens_neox': 3712, 'n_words': 1088, 'path': 'notebooks/MLpredictor.ipynb', 'repo_name': 'kant/watson-machine-learning-model-deploy', 'size': '12439'}