<jupyter_start><jupyter_text>AI Explanations: Explaining a tabular data model  OverviewIn this tutorial we will perform the following steps:1. Build and train a Keras model.1. Export the Keras model as a TF 1 SavedModel and deploy the model on Cloud AI Platform.1. Compute explainations for our model's predictions using Explainable AI on Cloud AI Platform.  DatasetThe dataset used for this tutorial was created from a BigQuery Public Dataset: [NYC 2018 Yellow Taxi data](https://console.cloud.google.com/bigquery?filter=solution-type:dataset&q=nyc%20taxi&id=e4902dee-0577-42a0-ac7c-436c04ea50b6&subtask=details&subtaskValue=city-of-new-york%2Fnyc-tlc-trips&project=michaelabel-gcp-training&authuser=1&subtaskIndex=3).   ObjectiveThe goal is to train a model using the Keras Sequential API that predicts how much a customer is compelled to pay (fares + tolls) for a taxi ride given the pickup location, dropoff location, the day of the week, and the hour of the day.This tutorial focuses more on deploying the model to AI Explanations than on the design of the model itself. We will be using preprocessed data for this lab. If you wish to know more about the data and how it was preprocessed please see this [notebook](https://github.com/GoogleCloudPlatform/training-data-analyst/blob/master/courses/machine_learning/deepdive/01_bigquery/c_extract_and_benchmark.ipynb).  Before you beginThis notebook was written with running in **Google Colabratory** in mind. The notebook will run on **Cloud AI Platform Notebooks** or your local environment if the proper packages are installed.Make sure you're running this notebook in a **GPU runtime** if you have that option. In Colab, select **Runtime** --> **Change runtime type** and select **GPU** for **Hardward Accelerator**.  Authenticate your GCP account**If you are using AI Platform Notebooks**, your environment is alreadyauthenticated. You should skip this step.**Be sure to change the `PROJECT_ID` below to your project before running the cell!**<jupyter_code>import os

PROJECT_ID = "michaelabel-gcp-training" 
os.environ["PROJECT_ID"] = PROJECT_ID<jupyter_output><empty_output><jupyter_text>**If you are using Colab**, run the cell below and follow the instructionswhen prompted to authenticate your account via oAuth. Ignore the error message related to `tensorflow-serving-api`.<jupyter_code>import sys
import warnings
warnings.filterwarnings('ignore')
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' 
# If you are running this notebook in Colab, follow the
# instructions to authenticate your GCP account. This provides access to your
# Cloud Storage bucket and lets you submit training jobs and prediction
# requests.

if 'google.colab' in sys.modules:
  from google.colab import auth as google_auth
  google_auth.authenticate_user()
  !pip install witwidget --quiet
  !pip install tensorflow==1.15.2 --quiet
  !gcloud config set project $PROJECT_ID

elif "DL_PATH" in os.environ:
  !sudo pip install tabulate --quiet<jupyter_output><empty_output><jupyter_text>Create a Cloud Storage bucket**The following steps are required, regardless of your notebook environment.**When you submit a training job using the Cloud SDK, you upload a Python packagecontaining your training code to a Cloud Storage bucket. AI Platform runsthe code from this package. In this tutorial, AI Platform also saves thetrained model that results from your job in the same bucket. You can thencreate an AI Platform model version based on this output in order to serveonline predictions.**Set the name of your Cloud Storage bucket below. It must be unique across allCloud Storage buckets.**You may also change the `REGION` variable, which is used for operationsthroughout the rest of this notebook. Make sure to [choose a region where CloudAI Platform services areavailable](https://cloud.google.com/ml-engine/docs/tensorflow/regions). Note that you maynot use a Multi-Regional Storage bucket for training with AI Platform.<jupyter_code>BUCKET_NAME = "michaelabel-gcp-training-ml" 
REGION = "us-central1"

os.environ['BUCKET_NAME'] = BUCKET_NAME
os.environ['REGION'] = REGION<jupyter_output><empty_output><jupyter_text>Run the following cell to create your Cloud Storage bucket if it does not already exist.<jupyter_code>%%bash
exists=$(gsutil ls -d | grep -w gs://${BUCKET_NAME}/)

if [ -n "$exists" ]; then
   echo -e "Bucket gs://${BUCKET_NAME} already exists."
    
else
   echo "Creating a new GCS bucket."
   gsutil mb -l ${REGION} gs://${BUCKET_NAME}
   echo -e "\nHere are your current buckets:"
   gsutil ls
fi<jupyter_output><empty_output><jupyter_text>Import libraries for creating modelImport the libraries we'll be using in this tutorial. **This tutorial has been tested with TensorFlow 1.15.2.**<jupyter_code>%tensorflow_version 1.x
import tensorflow as tf 
import tensorflow.feature_column as fc
import pandas as pd
import numpy as np 
import json
import time

# Should be 1.15.2
print(tf.__version__)<jupyter_output><empty_output><jupyter_text>Downloading and preprocessing dataIn this section you'll download the data to train and evaluate your model from a public GCS bucket. The original data has been preprocessed from the public BigQuery dataset linked above.<jupyter_code>%%bash
# Copy the data to your notebook instance
mkdir taxi_preproc
gsutil cp -r gs://cloud-training/bootcamps/serverlessml/taxi_preproc/*_xai.csv ./taxi_preproc
ls -l taxi_preproc<jupyter_output><empty_output><jupyter_text>Read the data with PandasWe'll use Pandas to read the training and validation data into a `DataFrame`. We will only use the first 7 columns of the csv files for our models.<jupyter_code>CSV_COLUMNS = ['fare_amount', 'dayofweek', 'hourofday', 'pickuplon',
             'pickuplat', 'dropofflon', 'dropofflat']

DAYS = ['Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat']
DTYPES = ['float32', 'str' , 'int32', 'float32' , 'float32' , 'float32' , 'float32' ]

def prepare_data(file_path):

  df = pd.read_csv(file_path, usecols = range(7), names = CSV_COLUMNS,
                   dtype = dict(zip(CSV_COLUMNS, DTYPES)), skiprows=1)
  
  labels = df['fare_amount'] 
  df = df.drop(columns=['fare_amount'])

  df['dayofweek'] = df['dayofweek'].map(dict(zip(DAYS, range(7)))).astype('float32')

  return df, labels

train_data, train_labels = prepare_data('./taxi_preproc/train_xai.csv')
valid_data, valid_labels = prepare_data('./taxi_preproc/valid_xai.csv')
# Preview the first 5 rows of training data
train_data.head()<jupyter_output><empty_output><jupyter_text>Build, train, and evaluate our model with Keras We'll use `tf.Keras` to build a our ML model that takes our features as input and predicts the fare amount. But first, we will do some feature engineering. We will be utilizing `tf.feature_column` and `tf.keras.layers.Lambda` to implement our feature engineering in the model graph to simplify our `serving_input_fn` later.<jupyter_code># Create functions to compute engineered features in later Lambda layers
def euclidean(params):
  lat1, lon1, lat2, lon2 = params
  londiff = lon2 - lon1
  latdiff = lat2 - lat1
  return tf.sqrt(londiff*londiff + latdiff*latdiff)
NUMERIC_COLS = ['pickuplon', 'pickuplat', 'dropofflon', 'dropofflat', 'hourofday', 'dayofweek']

def transform(inputs):

  transformed = inputs.copy()

  transformed['euclidean'] = tf.keras.layers.Lambda(euclidean, name='euclidean')([
              inputs['pickuplat'],
              inputs['pickuplon'],
              inputs['dropofflat'],
              inputs['dropofflon']])
  
  feat_cols = {colname: fc.numeric_column(colname)
           for colname in NUMERIC_COLS}

  feat_cols['euclidean'] = fc.numeric_column('euclidean')

  print("BEFORE TRANSFORMATION")
  print("INPUTS:", inputs.keys())

  print("AFTER TRANSFORMATION")
  print("TRANSFORMED:", transformed.keys())
  print("FEATURES", feat_cols.keys()) 

  return transformed, feat_cols

def build_model():

  raw_inputs = {
          colname : tf.keras.layers.Input(name=colname, shape=(), dtype='float32')
            for colname in NUMERIC_COLS
      }
  
  transformed, feat_cols = transform(raw_inputs)

  dense_inputs = tf.keras.layers.DenseFeatures(feat_cols.values(),
                                               name = 'dense_input')(transformed)

  h1 = tf.keras.layers.Dense(64, activation='relu', name='h1')(dense_inputs)
  h2 = tf.keras.layers.Dense(32, activation='relu', name='h2')(h1)
  output = tf.keras.layers.Dense(1, activation='linear', name = 'output')(h2)

  model = tf.keras.models.Model(raw_inputs, output)

  return model

model = build_model()
model.summary()
# Compile the model and see a summary
optimizer = tf.keras.optimizers.Adam(0.001)

model.compile(loss='mean_squared_error', optimizer=optimizer,
              metrics = [tf.keras.metrics.RootMeanSquaredError()])

tf.keras.utils.plot_model(model, to_file='model_plot.png', show_shapes=True, 
                          show_layer_names=True, rankdir="TB")<jupyter_output><empty_output><jupyter_text>Create an input data pipeline with tf.dataPer best practices, we will use `tf.Data` to create our input data pipeline. Our data is all in an in-memory dataframe, so we will use `tf.data.Dataset.from_tensor_slices` to create our pipeline.<jupyter_code>def load_dataset(features, labels, mode):

  dataset = tf.data.Dataset.from_tensor_slices(({"dayofweek" : features["dayofweek"],
                                                 "hourofday" : features["hourofday"],
                                                 "pickuplat" : features["pickuplat"],
                                                 "pickuplon" : features["pickuplon"],
                                                 "dropofflat" : features["dropofflat"],
                                                 "dropofflon" : features["dropofflon"]},
                                                  labels
                                                    ))


  if mode == tf.estimator.ModeKeys.TRAIN:
    dataset = dataset.repeat().batch(256).shuffle(256*10)
  else:
    dataset = dataset.batch(256)

  return dataset.prefetch(1)


train_dataset = load_dataset(train_data, train_labels, tf.estimator.ModeKeys.TRAIN)
valid_dataset = load_dataset(valid_data, valid_labels, tf.estimator.ModeKeys.EVAL)<jupyter_output><empty_output><jupyter_text>Train the modelNow we train the model. We will specify a number of epochs which to train the model and tell the model how many steps to expect per epoch.<jupyter_code>tf.keras.backend.get_session().run(tf.tables_initializer(name='init_all_tables'))

steps_per_epoch = 426433 // 256

model.fit(train_dataset, steps_per_epoch=steps_per_epoch, validation_data=valid_dataset, epochs=10)
# Send test instances to model for prediction
predict = model.predict(valid_dataset, steps = 1)
predict[:5]<jupyter_output><empty_output><jupyter_text>Export the model as a TF 1 SavedModelIn order to deploy our model in a format compatible with AI Explanations, we'll follow the steps below to convert our Keras model to a TF Estimator, and then use the `export_saved_model` method to generate the SavedModel and save it in GCS.<jupyter_code>## Convert our Keras model to an estimator
keras_estimator = tf.keras.estimator.model_to_estimator(keras_model=model, model_dir='export')
print(model.input)

# We need this serving input function to export our model in the next cell
serving_fn = tf.estimator.export.build_raw_serving_input_receiver_fn(
    model.input
)

export_path = keras_estimator.export_saved_model(
  'gs://' + BUCKET_NAME + '/explanations',
  serving_input_receiver_fn=serving_fn
).decode('utf-8')<jupyter_output><empty_output><jupyter_text>Use TensorFlow's `saved_model_cli` to inspect the model's SignatureDef. We'll use this information when we deploy our model to AI Explanations in the next section.<jupyter_code>!saved_model_cli show --dir $export_path --all<jupyter_output><empty_output><jupyter_text>Deploy the model to AI ExplanationsIn order to deploy the model to Explanations, we need to generate an `explanations_metadata.json` file and upload this to the Cloud Storage bucket with our SavedModel. Then we'll deploy the model using `gcloud`.  Prepare explanation metadataWe need to tell AI Explanations the names of the input and output tensors our model is expecting, which we print below. The value for `input_baselines` tells the explanations service what the baseline input should be for our model. Here we're using the median for all of our input features. That means the baseline prediction for this model will be the fare our model predicts for the median of each feature in our dataset.<jupyter_code># Print the names of our tensors
print('Model input tensors: ', model.input)
print('Model output tensor: ', model.output.name)
baselines_med = train_data.median().values.tolist()
baselines_mode = train_data.mode().values.tolist()
print(baselines_med)
print(baselines_mode)

explanation_metadata = {
    "inputs": {
      "dayofweek": {
        "input_tensor_name": "dayofweek:0",
        "input_baselines": [baselines_mode[0][0]] # Thursday
      },
      "hourofday": {
        "input_tensor_name": "hourofday:0",
        "input_baselines": [baselines_mode[0][1]] # 8pm
      },
      "dropofflon": {
        "input_tensor_name": "dropofflon:0",
        "input_baselines": [baselines_med[4]] 
      },
      "dropofflat": {
        "input_tensor_name": "dropofflat:0",
        "input_baselines": [baselines_med[5]] 
      },
      "pickuplon": {
        "input_tensor_name": "pickuplon:0",
        "input_baselines": [baselines_med[2]] 
      },
      "pickuplat": {
        "input_tensor_name": "pickuplat:0",
        "input_baselines": [baselines_med[3]] 
      },
    },
    "outputs": {
      "dense": {
        "output_tensor_name": "output/BiasAdd:0"
      }
    },
  "framework": "tensorflow"
  }

print(explanation_metadata)<jupyter_output><empty_output><jupyter_text>Since this is a regression model (predicting a numerical value), the baseline prediction will be the same for every example we send to the model. If this were instead a classification model, each class would have a different baseline prediction.<jupyter_code># Write the json to a local file
with open('explanation_metadata.json', 'w') as output_file:
  json.dump(explanation_metadata, output_file)
!gsutil cp explanation_metadata.json $export_path<jupyter_output><empty_output><jupyter_text>Create the modelNow we will create out model on Cloud AI Platform if it does not already exist.<jupyter_code>MODEL = 'taxifare_explain'
os.environ["MODEL"] = MODEL
%%bash
exists=$(gcloud ai-platform models list | grep ${MODEL})

if [ -n "$exists" ]; then
   echo -e "Model ${MODEL} already exists."
    
else
   echo "Creating a new model."
   gcloud ai-platform models create ${MODEL}
fi<jupyter_output><empty_output><jupyter_text>Create the model version Creating the version will take ~5-10 minutes. Note that your first deploy may take longer.<jupyter_code># Each time you create a version the name should be unique
import datetime
now = datetime.datetime.now().strftime("%Y%m%d%H%M%S")
VERSION_IG = 'v_IG_{}'.format(now)
VERSION_SHAP = 'v_SHAP_{}'.format(now)
# Create the version with gcloud
!gcloud beta ai-platform versions create $VERSION_IG \
--model $MODEL \
--origin $export_path \
--runtime-version 1.15 \
--framework TENSORFLOW \
--python-version 3.7 \
--machine-type n1-standard-4 \
--explanation-method 'integrated-gradients' \
--num-integral-steps 25

!gcloud beta ai-platform versions create $VERSION_SHAP \
--model $MODEL \
--origin $export_path \
--runtime-version 1.15 \
--framework TENSORFLOW \
--python-version 3.7 \
--machine-type n1-standard-4 \
--explanation-method 'sampled-shapley' \
--num-paths 50
# Make sure the model deployed correctly. State should be `READY` in the following log
!gcloud ai-platform versions describe $VERSION_IG --model $MODEL
!echo "---"
!gcloud ai-platform versions describe $VERSION_SHAP --model $MODEL<jupyter_output><empty_output><jupyter_text>Getting predictions and explanations on deployed modelNow that your model is deployed, you can use the AI Platform Prediction API to get feature attributions. We'll pass it a single test example here and see which features were most important in the model's prediction. Here we'll use `gcloud` to call our deployed model.  Format our request for gcloudTo use gcloud to make our AI Explanations request, we need to write the JSON to a file. Our example here is for a ride from the Google office in downtown Manhattan to LaGuardia Airport at 5pm on a Tuesday afternoon.Note that we had to write our day of the week at "3" instead of "Tue" since we encoded the days of the week outside of our model and serving input function.<jupyter_code># Format data for prediction to our model
!rm taxi-data.txt
!touch taxi-data.txt
prediction_json = {"dayofweek": "3", "hourofday": "17", "pickuplon": "-74.0026", "pickuplat": "40.7410", "dropofflat": "40.7790", "dropofflon": "-73.8772"}
with open('taxi-data.txt', 'a') as outfile:
  json.dump(prediction_json, outfile)
# Preview the contents of the data file
!cat taxi-data.txt<jupyter_output><empty_output><jupyter_text>Making the explain requestNow we make the explaination requests. We will go ahead and do this here for both integrated gradients and SHAP using the prediction JSON from above.<jupyter_code>resp_obj = !gcloud beta ai-platform explain --model $MODEL --version $VERSION_IG --json-instances='taxi-data.txt'
response_IG = json.loads(resp_obj.s)
resp_obj
resp_obj = !gcloud beta ai-platform explain --model $MODEL --version $VERSION_SHAP --json-instances='taxi-data.txt'
response_SHAP = json.loads(resp_obj.s)
resp_obj<jupyter_output><empty_output><jupyter_text>Understanding the explanations responseFirst let's just look at the difference between our predictions using our baselines and our predicted taxi fare for the example.<jupyter_code>explanations_IG = response_IG['explanations'][0]['attributions_by_label'][0]
explanations_SHAP = response_SHAP['explanations'][0]['attributions_by_label'][0]

predicted = round(explanations_SHAP['example_score'], 2)
baseline = round(explanations_SHAP['baseline_score'], 2 )
print('Baseline taxi fare: ' + str(baseline) + ' dollars')
print('Predicted taxi fare: ' + str(predicted) + ' dollars')<jupyter_output><empty_output><jupyter_text>Next let's look at the feature attributions for this particular example. Positive attribution values mean a particular feature pushed our model prediction up by that amount, and vice versa for negative attribution values. Which features seem like they're the most important...well it seems like the location features are the most important!<jupyter_code>from tabulate import tabulate

feature_names = valid_data.columns.tolist()
attributions_IG = explanations_IG['attributions']
attributions_SHAP = explanations_SHAP['attributions']
rows = []
for feat in feature_names:
  rows.append([feat, prediction_json[feat], attributions_IG[feat], attributions_SHAP[feat]])
print(tabulate(rows,headers=['Feature name', 'Feature value', 'Attribution value (IG)', 'Attribution value (SHAP)']))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09835636515912898, 'fraction_numerical': 0.010102596314907872, 'mean_word_length': 4.5296671490593345, 'pattern_counts': {'":': 29, '<': 81, '<?xml version=': 0, '>': 82, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '45509', 'license': 'Apache-2.0', 'n_tokens_mistral': 5902, 'n_tokens_neox': 5673, 'n_words': 1876, 'path': 'blogs/explainable_ai/AI_Explanations_on_CAIP.ipynb', 'repo_name': 'Glairly/introduction_to_tensorflow', 'size': '19104'}