<jupyter_start><jupyter_text>Generating Skewed Data for PredictionThis notebook helps generating skewed data based on the [covertype](https://archive.ics.uci.edu/ml/datasets/covertype) dataset from UCI Machine Learning Repository. The generated data is then used to simulate online prediction request workload to a deployed model version on the AI Platform Prediction.The notebook covers the following steps:1. Download the data2. Define dataset metadata3. Sample unskewed data points4. Prepare skewed data points5. Simulate serving workload to AI Platform Prediction  Setup  Install packages and dependencies<jupyter_code>!pip install -U -q google-api-python-client
!pip install -U -q pandas<jupyter_output><empty_output><jupyter_text>Setup your GCP Project<jupyter_code>PROJECT_ID = 'sa-data-validation'
BUCKET =  'sa-data-validation'
REGION = 'us-central1'
!gcloud config set project $PROJECT_ID<jupyter_output><empty_output><jupyter_text>Authenticate your GCP accountThis is required if you run the notebook in Colab<jupyter_code>try:
  from google.colab import auth
  auth.authenticate_user()
  print("Colab user is authenticated.")
except: pass<jupyter_output><empty_output><jupyter_text>Import libraries<jupyter_code>import os
from tensorflow import io as tf_io
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np<jupyter_output><empty_output><jupyter_text>Define constantsYou can change the default values for the following constants<jupyter_code>LOCAL_WORKSPACE = './workspace'
LOCAL_DATA_DIR = os.path.join(LOCAL_WORKSPACE, 'data')
LOCAL_DATA_FILE = os.path.join(LOCAL_DATA_DIR, 'train.csv')
BQ_DATASET_NAME = 'data_validation'
BQ_TABLE_NAME = 'covertype_classifier_logs'
MODEL_NAME = 'covertype_classifier'
VERSION_NAME = 'v1'
MODEL_OUTPUT_KEY = 'probabilities'
SIGNATURE_NAME = 'serving_default'<jupyter_output><empty_output><jupyter_text>1. Download DataThe covertype dataset is preprocessed, split, and uploaded to uploaded to the `gs://workshop-datasets/covertype` public GCS location. We use this version of the preprocessed dataset in this notebook. For more information, see [Cover Type Dataset](https://github.com/GoogleCloudPlatform/mlops-on-gcp/tree/master/datasets/covertype)<jupyter_code>if tf_io.gfile.exists(LOCAL_WORKSPACE):
  print("Removing previous workspace artifacts...")
  tf_io.gfile.rmtree(LOCAL_WORKSPACE)

print("Creating a new workspace...")
tf_io.gfile.makedirs(LOCAL_WORKSPACE)
tf_io.gfile.makedirs(LOCAL_DATA_DIR)
!gsutil cp gs://workshop-datasets/covertype/data_validation/training/dataset.csv {LOCAL_DATA_FILE}
!wc -l {LOCAL_DATA_FILE}
data = pd.read_csv(LOCAL_DATA_FILE)
print("Total number of records: {}".format(len(data.index)))
data.sample(10).T<jupyter_output><empty_output><jupyter_text>2. Define Metadata<jupyter_code>HEADER = ['Elevation', 'Aspect', 'Slope','Horizontal_Distance_To_Hydrology',
          'Vertical_Distance_To_Hydrology', 'Horizontal_Distance_To_Roadways',
          'Hillshade_9am', 'Hillshade_Noon', 'Hillshade_3pm',
          'Horizontal_Distance_To_Fire_Points', 'Wilderness_Area', 'Soil_Type',
          'Cover_Type']

TARGET_FEATURE_NAME = 'Cover_Type'

FEATURE_LABELS = ['0', '1', '2', '3', '4', '5', '6']

NUMERIC_FEATURE_NAMES = ['Aspect', 'Elevation', 'Hillshade_3pm', 
                         'Hillshade_9am', 'Hillshade_Noon', 
                         'Horizontal_Distance_To_Fire_Points',
                         'Horizontal_Distance_To_Hydrology',
                         'Horizontal_Distance_To_Roadways','Slope',
                         'Vertical_Distance_To_Hydrology']

CATEGORICAL_FEATURE_NAMES = ['Soil_Type', 'Wilderness_Area']

FEATURE_NAMES = CATEGORICAL_FEATURE_NAMES + NUMERIC_FEATURE_NAMES

HEADER_DEFAULTS = [[0] if feature_name in NUMERIC_FEATURE_NAMES + [TARGET_FEATURE_NAME] else ['NA'] 
                   for feature_name in HEADER]

NUM_CLASSES = len(FEATURE_LABELS)
for feature_name in CATEGORICAL_FEATURE_NAMES:
  data[feature_name] = data[feature_name].astype(str)<jupyter_output><empty_output><jupyter_text>3. Sampling Normal Data<jupyter_code>normal_data = data.sample(2000)
fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(20, 10))
normal_data['Elevation'].plot.hist(bins=15, ax=axes[0][0], title='Elevation')
normal_data['Aspect'].plot.hist(bins=15, ax=axes[0][1], title='Aspect')
normal_data['Wilderness_Area'].value_counts(normalize=True).plot.bar(ax=axes[1][0], title='Wilderness Area')
normal_data[TARGET_FEATURE_NAME].value_counts(normalize=True).plot.bar(ax=axes[1][1], title=TARGET_FEATURE_NAME)<jupyter_output><empty_output><jupyter_text>4. Prepare Skewed DataWe are going to introduce the following skews to the data:1. **Numerical Features** * *Elevation - Feature Skew*: Convert the unit of measure from meters to kilometers for 1% of the data points * *Aspect - Distribution Skew*: Decrease the value by randomly from 1% to 50%2. **Categorical Features** * *Wilderness_Area - Feature Skew*: Adding a new category "Others" for 1% of the data points * *Wilderness_Area - Distribution Skew*: Increase of of the frequency of "Cache" and "Neota" values by 25%<jupyter_code>skewed_data = data.sample(1000)<jupyter_output><empty_output><jupyter_text>4.1 Skewing numerical features  4.1.1 Elevation Feature Skew<jupyter_code>ratio = 0.1
size = int(len(skewed_data.index) * ratio)
indexes = np.random.choice(skewed_data.index, size=size, replace=False)
skewed_data['Elevation'][indexes] = skewed_data['Elevation'][indexes] // 1000
fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(20, 5))
normal_data['Elevation'].plot.hist(bins=15, ax=axes[0], title='Elevation - Normal')
skewed_data['Elevation'].plot.hist(bins=15,  ax=axes[1], title='Elevation - Skewed')<jupyter_output><empty_output><jupyter_text>4.1.2 Aspect Distribution Skew<jupyter_code>skewed_data['Aspect'] = skewed_data['Aspect'].apply(
    lambda value: int(value * np.random.uniform(0.5, 0.99))
)
fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(20, 5))
normal_data['Aspect'].plot.hist(bins=15, ax=axes[0], title='Aspect - Normal')
skewed_data['Aspect'].plot.hist(bins=15,  ax=axes[1], title='Aspect - Skewed')<jupyter_output><empty_output><jupyter_text>4.2 Skew categorical features  4.2.1 Wilderness Area Feature SkewAdding a new category "Others"<jupyter_code>skewed_data['Wilderness_Area'] = skewed_data['Wilderness_Area'].apply(
    lambda value: 'Others' if np.random.uniform() <= 0.1 else value
)<jupyter_output><empty_output><jupyter_text>4.2.2 Wilderness Area Distribution Skew<jupyter_code>skewed_data['Wilderness_Area'] = skewed_data['Wilderness_Area'].apply(
    lambda value: 'Neota' if value in ['Rawah', 'Commanche'] and np.random.uniform() <= 0.25 else value
)
fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(20, 5))
normal_data['Wilderness_Area'].value_counts(normalize=True).plot.bar(ax=axes[0], title='Wilderness Area - Normal')
skewed_data['Wilderness_Area'].value_counts(normalize=True).plot.bar(ax=axes[1], title='Wilderness Area - Skewed')<jupyter_output><empty_output><jupyter_text>5. Simulating serving workload  5.1 Implement the model API client<jupyter_code>import googleapiclient.discovery
import numpy as np

service = googleapiclient.discovery.build('ml', 'v1')
name = 'projects/{}/models/{}/versions/{}'.format(PROJECT_ID, MODEL_NAME, VERSION_NAME)
print("Service name: {}".format(name))

def caip_predict(instance):
  
  request_body={
      'signature_name': SIGNATURE_NAME,
      'instances': [instance]
      }

  response = service.projects().predict(
      name=name,
      body=request_body

  ).execute()

  if 'error' in response:
    raise RuntimeError(response['error'])

  probability_list = [output[MODEL_OUTPUT_KEY] for output in response['predictions']]
  classes = [FEATURE_LABELS[int(np.argmax(probabilities))] for probabilities in probability_list]
  return classes
import time

def simulate_requests(data_frame):

  print("Simulation started...")
  print("---------------------")
  print("Number of instances: {}".format(len(data_frame.index)))

  i = 0
  for _, row in data_frame.iterrows():
    instance = dict(row)
    instance.pop(TARGET_FEATURE_NAME)
    for k,v in instance.items():
      instance[k] = [v]

    predicted_class = caip_predict(instance)
    i += 1
    
    print(".", end='')

    if (i + 1) % 100 == 0:
      print()
      print("Sent {} requests.".format(i + 1))

    time.sleep(0.5)
  print("")
  print("-------------------")
  print("Simulation finised.")<jupyter_output><empty_output><jupyter_text>5.2 Simulate AI Platform Prediction requests<jupyter_code>simulate_requests(normal_data)
simulate_requests(skewed_data)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1280982163539495, 'fraction_numerical': 0.01586750057910586, 'mean_word_length': 5.788522012578617, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 61, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '17888', 'license': 'Apache-2.0', 'n_tokens_mistral': 3040, 'n_tokens_neox': 2902, 'n_words': 605, 'path': 'skew_detection/workload_simulator/covertype_data_generation.ipynb', 'repo_name': 'gitgithan/mlops-on-gcp', 'size': '8634'}