<jupyter_start><jupyter_text>Use Tensorflow to recognize hand-written digits with `ibm-watson-machine-learning` This notebook contains the steps and code to demonstrate support of Deep Learning model training and scoring in the Watson Machine Learning service. It introduces commands for data retrieval, `training_definition` persistance to Watson Machine Learning repository, model training, model persistance, model deployment and scoring.Some familiarity with Python is helpful. This notebook uses Python 3.8. Learning goalsThe learning goals of this notebook are:-  Working with Watson Machine Learning service.-  Training Deep Learning models (TensorFlow).-  Saving trained models in Watson Machine Learning repository.-  Online deployment and scoring of trained model. ContentsThis notebook contains the following parts:1.	[Setup](setup)2.	[Create model definition](model_def)3.	[Train model](training)4.  [Persist trained model](persist)5.	[Deploy and Score](deploy)6.  [Clean up](clean)7.	[Summary and next steps](summary)  1. Set up the environmentBefore you use the sample code in this notebook, you must perform the following setup tasks:-  Contact with your Cloud Pack for Data administrator and ask him for your account credentials  Connection to WMLAuthenticate the Watson Machine Learning service on IBM Cloud Pack for Data. You need to provide platform `url`, your `username` and `api_key`.<jupyter_code>username = 'PASTE YOUR USERNAME HERE'
api_key = 'PASTE YOUR API_KEY HERE'
url = 'PASTE THE PLATFORM URL HERE'
wml_credentials = {
    "username": username,
    "apikey": api_key,
    "url": url,
    "instance_id": 'openshift',
    "version": '4.0'
}<jupyter_output><empty_output><jupyter_text>Alternatively you can use `username` and `password` to authenticate WML services.```wml_credentials = {    "username": ***,    "password": ***,    "url": ***,    "instance_id": 'openshift',    "version": '4.0'}```  Install and import the `ibm-watson-machine-learning` package**Note:** `ibm-watson-machine-learning` documentation can be found here.<jupyter_code>!pip install -U ibm-watson-machine-learning
from ibm_watson_machine_learning import APIClient

client = APIClient(wml_credentials)<jupyter_output><empty_output><jupyter_text>Working with spacesFirst of all, you need to create a space that will be used for your work. If you do not have space already created, you can use `{PLATFORM_URL}/ml-runtime/spaces?context=icp4data` to create one.- Click New Deployment Space- Create an empty space- Go to space `Settings` tab- Copy `space_id` and paste it below**Tip**: You can also use SDK to prepare the space for your work. More information can be found [here](https://github.com/IBM/watson-machine-learning-samples/blob/master/cpd4.0/notebooks/python_sdk/instance-management/Space%20management.ipynb).**Action**: Assign space ID below<jupyter_code>space_id = 'PASTE YOUR SPACE ID HERE'<jupyter_output><empty_output><jupyter_text>You can use `list` method to print all existing spaces.<jupyter_code>client.spaces.list(limit=10)<jupyter_output><empty_output><jupyter_text>To be able to interact with all resources available in Watson Machine Learning, you need to set **space** which you will be using.<jupyter_code>client.set.default_space(space_id)<jupyter_output><empty_output><jupyter_text>2. Create model definition  2.1 Prepare model definition metadata<jupyter_code>model_definition_metadata = {
    client.model_definitions.ConfigurationMetaNames.NAME: "Hand-written Digit Recognition",
    client.model_definitions.ConfigurationMetaNames.DESCRIPTION: "Hand-written Digit Recognition",
    client.model_definitions.ConfigurationMetaNames.COMMAND:  "convolutional_network.py --trainImagesFile train-images-idx3-ubyte.gz --trainLabelsFile train-labels-idx1-ubyte.gz --testImagesFile t10k-images-idx3-ubyte.gz --testLabelsFile t10k-labels-idx1-ubyte.gz --learningRate 0.001 --trainingIters 200",
    client.model_definitions.ConfigurationMetaNames.PLATFORM: {"name": "python", "versions": ["3.8"]},
    client.model_definitions.ConfigurationMetaNames.VERSION: "2.0",
    client.model_definitions.ConfigurationMetaNames.SPACE_UID: space_id
}<jupyter_output><empty_output><jupyter_text>2.2  Get sample model definition content file from git**Hint:** You may need to install the `wget` using following command `!pip install wget`<jupyter_code>import os, wget

filename='tf-softmax-model.zip'

if not os.path.isfile(filename):
    filename = wget.download('https://github.com/IBM/watson-machine-learning-samples/raw/master/cloud/definitions/tensorflow/tf-softmax-model.zip')<jupyter_output><empty_output><jupyter_text>**Tip**: Convert below cell to code and run it to see model deinition's code.<jupyter_code>!unzip -oqd . tf-softmax-model.zip && cat convolutional_network.py<jupyter_output><empty_output><jupyter_text>2.3  Publish model definition<jupyter_code>definition_details = client.model_definitions.store(filename, model_definition_metadata)
model_definition_id = client.model_definitions.get_id(definition_details)
print(model_definition_id)
client.model_definitions.get_details(model_definition_id)<jupyter_output><empty_output><jupyter_text>List model definitions<jupyter_code>client.model_definitions.list(limit=5)<jupyter_output><empty_output><jupyter_text>3. Train model  **Warning**: Before executing deep learning experiment make sure that [training data](https://github.com/IBM/watson-machine-learning-samples/tree/master/cpd4.0/data/mnist/raw) is saved in a folder where Watson Machine Learning Accelerator is installed.  3.1 Prepare training metadata<jupyter_code>training_metadata = training_metadata = {
   client.training.ConfigurationMetaNames.NAME: "Hand-written Digit Recognition",
    client.training.ConfigurationMetaNames.SPACE_UID: space_id,
    client.training.ConfigurationMetaNames.DESCRIPTION: "Hand-written Digit Recognition",
    client.training.ConfigurationMetaNames.TRAINING_RESULTS_REFERENCE: {
      "name":"MNIST results",
      "connection":{},
      "location":{
         "path":f"spaces/{space_id}/assets/experiment"
      },
      "type":"fs"
   },
  client.training.ConfigurationMetaNames.MODEL_DEFINITION:{
        "id": model_definition_id,
        "hardware_spec": {
          "name": "K80",
          "nodes": 1
        },
        "software_spec": {
          "name": "tensorflow_2.4-py3.8"
        }
  },
  client.training.ConfigurationMetaNames.TRAINING_DATA_REFERENCES: [
      {
         "name":"training_input_data",
         "type":"fs",
         "connection":{},
         "location":{
            "path": "tf-mnist"
         },
         "schema":{
            "id":"idmlp_schema",
            "fields":[
               {
                  "name":"text",
                  "type":"string"
               }
            ]
         }
      }
   ]
}<jupyter_output><empty_output><jupyter_text>3.2 Train model in background<jupyter_code>training = client.training.run(training_metadata)<jupyter_output><empty_output><jupyter_text>3.3 Get training id and status<jupyter_code>training_id = client.training.get_id(training)
client.training.get_status(training_id)["state"]<jupyter_output><empty_output><jupyter_text>3.4 Get training details<jupyter_code>import json

training_details = client.training.get_details(training_id)
print(json.dumps(training_details, indent=2))<jupyter_output><empty_output><jupyter_text>List trainings<jupyter_code>client.training.list(limit=5)<jupyter_output><empty_output><jupyter_text>Cancel training You can cancel the training run by calling the method below.  **Tip**: If you want to delete train runs and results, add `hard_delete=True` as a parameter.<jupyter_code>client.training.cancel(training_id)<jupyter_output><empty_output><jupyter_text>4. Persist trained model  4.1 Publish model<jupyter_code>software_spec_uid = client.software_specifications.get_uid_by_name('tensorflow_2.4-py3.8')
model_meta_props = {client.repository.ModelMetaNames.NAME: "TF Mnist Model",
                    client.repository.ModelMetaNames.TYPE: "tensorflow_2.4",
                    client.repository.ModelMetaNames.SOFTWARE_SPEC_UID: software_spec_uid
                    }

published_model_details = client.repository.store_model(training_id, meta_props=model_meta_props)
model_uid = client.repository.get_model_uid(published_model_details)<jupyter_output><empty_output><jupyter_text>4.2 Get model details<jupyter_code>model_details = client.repository.get_details(model_uid)
print(json.dumps(model_details, indent=2))<jupyter_output>{
  "entity": {
    "content_import_state": "completed",
    "model_definition": {
      "id": "9b9ca252-7615-4b52-be79-e993629a57f6"
    },
    "software_spec": {
      "id": "fe185c44-9a99-5425-986b-59bd1d2eda46",
      "name": "tensorflow_2.4-py3.8"
    },
    "type": "tensorflow_2.4"
  },
  "metadata": {
    "created_at": "2021-12-08T12:58:39.929Z",
    "id": "4c362397-3d08-48a7-93b5-1599d22fd380",
    "modified_at": "2021-12-08T12:58:43.686Z",
    "name": "TF Mnist Model",
    "owner": "1000330999",
    "resource_key": "23de2ef7-4afa-4869-8c59-d95f69be814b",
    "space_id": "c4f8dcc6-2f39-4a7d-8c85-cda634f8c52c"
  },
  "system": {
    "warnings": []
  }
}<jupyter_text>List stored models<jupyter_code>client.repository.list_models(limit=5)<jupyter_output><empty_output><jupyter_text>5. Deploy and score  5.1 Create online deployment for published model You can deploy the stored model as a web service (online) by running the code in the following cell.<jupyter_code>deployment = client.deployments.create(model_uid, meta_props={
                                            client.deployments.ConfigurationMetaNames.NAME:"TF Mnist deployment",
                                            client.deployments.ConfigurationMetaNames.ONLINE:{}})

deployment_uid = client.deployments.get_uid(deployment)<jupyter_output>#######################################################################################

Synchronous deployment creation for uid: '4c362397-3d08-48a7-93b5-1599d22fd380' started

#######################################################################################


initializingNote: online_url is deprecated and will be removed in a future release. Use serving_urls instead.

Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead.
.Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead.
.Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead.
.Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead.

ready


------------------------------------------------------------------------------------------------
Successfully finished deployment creation, deployment_uid='273eccbd-849c-429d-a491-82edd32b068e'
---------------------[...]<jupyter_text>5.2 Get deployments details<jupyter_code>deployments_details = client.deployments.get_details(deployment_uid)
print(json.dumps(deployments_details, indent=2))<jupyter_output><empty_output><jupyter_text>List deployments<jupyter_code>client.deployments.list(limit=5)<jupyter_output><empty_output><jupyter_text>5.3 Score deployed model Prepare sample scoring data to score deployed model.<jupyter_code>import wget

dataset_filename='mnist.npz'

if not os.path.isfile(dataset_filename):
    dataset_filename = wget.download('https://github.com/IBM/watson-machine-learning-samples/raw/master/cpd4.0/data/mnist/mnist.npz')
import numpy as np

mnist_dataset = np.load(dataset_filename)
x_test = mnist_dataset['x_test']
image_1 = x_test[0].ravel() / 255
image_2 = x_test[1].ravel() / 255
%matplotlib inline
import matplotlib.pyplot as plt
for i, image in enumerate([x_test[0], x_test[1]]):
    plt.subplot(2, 2, i + 1)
    plt.axis('off')
    plt.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')<jupyter_output><empty_output><jupyter_text>Build scoring ditionary consisting of two digits and send it to deployed model to get predictions.<jupyter_code>scoring_payload = {
    client.deployments.ScoringMetaNames.INPUT_DATA : [
        {'values': [image_1.tolist(), image_2.tolist()]}
    ]
}
scores = client.deployments.score(deployment_uid, meta_props=scoring_payload)
print("Scoring result:\n" + json.dumps(scores, indent=2))<jupyter_output>Scoring result:
{
  "predictions": [
    {
      "id": "classes",
      "values": [
        7,
        2
      ]
    }
  ]
}
{'fraction_non_alphanumeric': 0.13003618817852836, 'fraction_numerical': 0.02501005227181343, 'mean_word_length': 4.738809413936317, 'pattern_counts': {'":': 54, '<': 94, '<?xml version=': 0, '>': 94, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '136199', 'license': 'Apache-2.0', 'n_tokens_mistral': 4048, 'n_tokens_neox': 3825, 'n_words': 849, 'path': 'cpd4.0/notebooks/python_sdk/experiments/deep_learning/Use TensorFlow to recognize hand-written digits.ipynb', 'repo_name': 'muthukumarbala07/watson-machine-learning-samples', 'size': '12435'}