<jupyter_start><jupyter_text>Introducing the Keras Sequential API**Learning Objectives**  1. Build a DNN model using the Keras Sequential API  1. Learn how to use feature columns in a Keras model  1. Learn how to train a model with Keras  1. Learn how to save/load, and deploy a Keras model on GCP  1. Learn how to deploy and make predictions with at Keras model IntroductionThe [Keras sequential API](https://keras.io/models/sequential/) allows you to create Tensorflow models layer-by-layer. This is useful for building most kinds of machine learning models but it does not allow you to create models that share layers, re-use layers or have multiple inputs or outputs. In this lab, we'll see how to build a simple deep neural network model using the keras sequential api and feature columns. Once we have trained our model, we will deploy it using AI Platform and see how to call our model for online prediciton. Start by importing the necessary libraries for this lab.<jupyter_code>import datetime
import os
import shutil

import numpy as np
import pandas as pd
import tensorflow as tf

from matplotlib import pyplot as plt
from tensorflow import keras

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, DenseFeatures
from tensorflow.keras.callbacks import TensorBoard

print(tf.__version__)
%matplotlib inline<jupyter_output>2.1.1<jupyter_text>Load raw data We will use the taxifare dataset, using the CSV files that we created in the first notebook of this sequence. Those files have been saved into `../data`.<jupyter_code>!ls -l ../data/*.csv
!head ../data/taxi*.csv<jupyter_output>==> ../data/taxi-test.csv <==
6.0,2013-03-27 03:35:00 UTC,-73.977672,40.784052,-73.965332,40.801025,2,0
19.3,2012-05-10 18:43:16 UTC,-73.954366,40.778924,-74.004094,40.723104,1,1
7.5,2014-05-20 23:09:00 UTC,-73.999165,40.738377,-74.003473,40.723862,2,2
12.5,2015-02-23 19:51:31 UTC,-73.9652099609375,40.76948165893555,-73.98949432373047,40.739742279052734,1,3
10.9,2011-03-19 03:32:00 UTC,-73.99259,40.742957,-73.989908,40.711053,1,4
7.0,2012-09-18 12:51:11 UTC,-73.971195,40.751566,-73.975922,40.756361,1,5
19.0,2014-05-20 23:09:00 UTC,-73.998392,40.74517,-73.939845,40.74908,1,6
8.9,2012-07-18 08:46:08 UTC,-73.997638,40.756541,-73.973303,40.762019,1,7
4.5,2010-07-11 20:39:08 UTC,-73.976738,40.751321,-73.986671,40.74883,1,8
7.0,2013-12-12 02:16:40 UTC,-73.985024,40.767537,-73.981273,40.779302,1,9

==> ../data/taxi-train.csv <==
11.3,2011-01-28 20:42:59 UTC,-73.999022,40.739146,-73.990369,40.717866,1,0
7.7,2011-06-27 04:28:06 UTC,-73.987443,40.729221,-73.979013,40.758641,1,1
10.5,2011-04-03 0[...]<jupyter_text>Use tf.data to read the CSV filesWe wrote these functions for reading data from the csv files above in the [previous notebook](https://github.com/GoogleCloudPlatform/training-data-analyst/blob/master/courses/machine_learning/deepdive2/introduction_to_tensorflow/labs/2_dataset_api.ipynb).<jupyter_code>CSV_COLUMNS = [
    'fare_amount',
    'pickup_datetime',
    'pickup_longitude',
    'pickup_latitude',
    'dropoff_longitude',
    'dropoff_latitude',
    'passenger_count',
    'key'
]
LABEL_COLUMN = 'fare_amount'
DEFAULTS = [[0.0], ['na'], [0.0], [0.0], [0.0], [0.0], [0.0], ['na']]
UNWANTED_COLS = ['pickup_datetime', 'key']


def features_and_labels(row_data):
    label = row_data.pop(LABEL_COLUMN)
    features = row_data
    
    for unwanted_col in UNWANTED_COLS:
        features.pop(unwanted_col)

    return features, label


def create_dataset(pattern, batch_size=1, mode='eval'):
    dataset = tf.data.experimental.make_csv_dataset(
        pattern, batch_size, CSV_COLUMNS, DEFAULTS) #JOSE: the pattern is a file path

    dataset = dataset.map(features_and_labels)

    if mode == 'train':
        dataset = dataset.shuffle(buffer_size=1000).repeat()

    # take advantage of multi-threading; 1=AUTOTUNE
    dataset = dataset.prefetch(1)
    return dataset<jupyter_output><empty_output><jupyter_text>Build a simple keras DNN modelWe will use feature columns to connect our raw data to our keras DNN model. Feature columns make it easy to perform common types of feature engineering on your raw data. For example, you can one-hot encode categorical data, create feature crosses, embeddings and more. We'll cover these in more detail later in the course, but if you want to a sneak peak browse the official TensorFlow [feature columns guide](https://www.tensorflow.org/guide/feature_columns).In our case we won't do any feature engineering. However, we still need to create a list of feature columns to specify the numeric values which will be passed on to our model. To do this, we use `tf.feature_column.numeric_column()`We use a python dictionary comprehension to create the feature columns for our model, which is just an elegant alternative to a for loop. **Exercise.** Create a feature column dictionary that we will use when building our deep neural network below. The keys should be the element of the `INPUT_COLS` list, while the values should be numeric feature columns.<jupyter_code>INPUT_COLS = [
    'pickup_longitude',
    'pickup_latitude',
    'dropoff_longitude',
    'dropoff_latitude',
    'passenger_count',
]

# Create input layer of feature columns

feature_columns = {
colname: tf.feature_column.numeric_column(colname)
for colname in INPUT_COLS}
#    tf.feature_column.numeric_column('pickup_longitude'),
#    tf.feature_column.numeric_column('pickup_latitude'),
#    tf.feature_column.numeric_column('dropoff_longitude'),
#    tf.feature_column.numeric_column('dropoff_latitude'),
#    tf.feature_column.numeric_column('passenger_count')
#] # TODO: Your code here<jupyter_output><empty_output><jupyter_text>Next, we create the DNN model. The Sequential model is a linear stack of layers and when building a model using the Sequential API, you configure each layer of the model in turn. Once all the layers have been added, you compile the model.  **Exercise.** Create a deep neural network using Keras's Sequential API. In the cell below, use the `tf.keras.layers` library to create all the layers for your deep neural network.<jupyter_code># Build a keras DNN model using Sequential API
model = Sequential([
    DenseFeatures(feature_columns=feature_columns.values()),
    Dense(units=30, activation='relu'),
    Dense(units=10, activation='relu'),
    Dense(units=1, activation='linear'),
]) # TODO: Your code here ESTO LO TENÍA MASL POR LA SINTAXIS: yo ponía tf.keras.dense...<jupyter_output><empty_output><jupyter_text>Next, to prepare the model for training, you must configure the learning process. This is done using the compile method. The compile method takes three arguments:* An optimizer. This could be the string identifier of an existing optimizer (such as `rmsprop` or `adagrad`), or an instance of the [Optimizer class](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/optimizers).* A loss function. This is the objective that the model will try to minimize. It can be the string identifier of an existing loss function from the [Losses class](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/losses) (such as categorical_crossentropy or mse), or it can be a custom objective function.* A list of metrics. For any machine learning problem you will want a set of metrics to evaluate your model. A metric could be the string identifier of an existing metric or a custom metric function.We will add an additional custom metric called `rmse` to our list of metrics which will return the root mean square error.  **Exercise.** Compile the model you created above. Create a custom loss function called `rmse` which computes the root mean squared error between `y_true` and `y_pred`. Pass this function to the model as an evaluation metric.<jupyter_code># Create a custom evalution metric
def rmse(y_true, y_pred):
    return tf.sqrt(tf.reduce_mean(tf.square(y_pred-y_true))) # TODO: Your code here

# Compile the keras model
model.compile(optimizer="adam", loss="mse", metrics=[rmse, "mse"]) # TODO: Your code here<jupyter_output><empty_output><jupyter_text>Train the modelTo train your model, Keras provides three functions that can be used: 1. `.fit()` for training a model for a fixed number of epochs (iterations on a dataset). 2. `.fit_generator()` for training a model on data yielded batch-by-batch by a generator 3. `.train_on_batch()` runs a single gradient update on a single batch of data.  The `.fit()` function works well for small datasets which can fit entirely in memory. However, for large datasets (or if you need to manipulate the training data on the fly via data augmentation, etc) you will need to use `.fit_generator()` instead. The `.train_on_batch()` method is for more fine-grained control over training and accepts only a single batch of data.The taxifare dataset we sampled is small enough to fit in memory, so can we could use `.fit` to train our model. Our `create_dataset` function above generates batches of training examples, so we could also use `.fit_generator`. In fact, when calling `.fit` the method inspects the data, and if it's a generator (as our dataset is) it will invoke automatically `.fit_generator` for training. We start by setting up some parameters for our training job and create the data generators for the training and validation data.We refer you the the blog post [ML Design Pattern 3: Virtual Epochs](https://medium.com/google-cloud/ml-design-pattern-3-virtual-epochs-f842296de730) for further details on why express the training in terms of `NUM_TRAIN_EXAMPLES` and `NUM_EVALS` and why, in this training code, the number of epochs is really equal to the number of evaluations we perform.<jupyter_code>TRAIN_BATCH_SIZE = 1000
NUM_TRAIN_EXAMPLES = 10000 * 5  # training dataset will repeat, wrap around
NUM_EVALS = 50  # how many times to evaluate
NUM_EVAL_EXAMPLES = 10000  # enough to get a reasonable sample

trainds = create_dataset(
    pattern='../data/taxi-train*',
    batch_size=TRAIN_BATCH_SIZE,
    mode='train')

evalds = create_dataset(
    pattern='../data/taxi-valid*',
    batch_size=1000,
    mode='eval').take(NUM_EVAL_EXAMPLES//1000)<jupyter_output><empty_output><jupyter_text>There are various arguments you can set when calling the [.fit method](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/Modelfit). Here `x` specifies the input data which in our case is a `tf.data` dataset returning a tuple of (inputs, targets). The `steps_per_epoch` parameter is used to mark the end of training for a single epoch. Here we are training for NUM_EVALS epochs. Lastly, for the `callback` argument we specify a Tensorboard callback so we can inspect Tensorboard after training.  **Exercise.** In the cell below, you will train your model. First, define the `steps_per_epoch` then train your model using `.fit()`, saving the model training output to a variable called `history`.<jupyter_code>%time 
steps_per_epoch = NUM_TRAIN_EXAMPLES // (TRAIN_BATCH_SIZE * NUM_EVALS) # TODO: Your code here 

LOGDIR = "./taxi_trained"
history = model.fit(x=trainds,
    steps_per_epoch=steps_per_epoch,
    epochs=NUM_EVALS,
    validation_data=evalds,
    callbacks=[TensorBoard(LOGDIR)]
)
# TODO: Your code here<jupyter_output>CPU times: user 3 µs, sys: 0 ns, total: 3 µs
Wall time: 5.96 µs
Train for 1 steps, validate for 10 steps
Epoch 1/50
1/1 [==============================] - 16s 16s/step - loss: 2596.7300 - rmse: 50.9581 - mse: 2596.7300 - val_loss: 2525.5294 - val_rmse: 50.2532 - val_mse: 2525.5293
Epoch 2/50
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (0.413867). Check your callbacks.
1/1 [==============================] - 1s 828ms/step - loss: 2420.7737 - rmse: 49.2014 - mse: 2420.7737 - val_loss: 2362.9251 - val_rmse: 48.6089 - val_mse: 2362.9250
Epoch 3/50
1/1 [==============================] - 0s 440ms/step - loss: 2276.4104 - rmse: 47.7117 - mse: 2276.4104 - val_loss: 2201.6777 - val_rmse: 46.9201 - val_mse: 2201.6775
Epoch 4/50
1/1 [==============================] - 0s 387ms/step - loss: 2137.9326 - rmse: 46.2378 - mse: 2137.9326 - val_loss: 2054.4463 - val_rmse: 45.3251 - val_mse: 2054.4463
Epoch 5/50
1/1 [==============================] - 0s 384ms/step - [...]<jupyter_text>High-level model evaluationOnce we've run data through the model, we can call `.summary()` on the model to get a high-level summary of our network. We can also plot the training and evaluation curves for the metrics we computed above.<jupyter_code>model.summary()<jupyter_output>Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_features_1 (DenseFeatu multiple                  0         
_________________________________________________________________
dense_3 (Dense)              multiple                  180       
_________________________________________________________________
dense_4 (Dense)              multiple                  310       
_________________________________________________________________
dense_5 (Dense)              multiple                  11        
=================================================================
Total params: 501
Trainable params: 501
Non-trainable params: 0
_________________________________________________________________<jupyter_text>Running `.fit` (or `.fit_generator`) returns a History object which collects all the events recorded during training. Similar to Tensorboard, we can plot the training and validation curves for the model loss and rmse by accessing these elements of the History object.<jupyter_code>RMSE_COLS = ['rmse', 'val_rmse']

pd.DataFrame(history.history)[RMSE_COLS].plot() #JOSE: history = output of training
LOSS_COLS = ['loss', 'val_loss']

pd.DataFrame(history.history)[LOSS_COLS].plot()<jupyter_output><empty_output><jupyter_text>Making predictions with our modelTo make predictions with our trained model, we can call the [predict method](https://www.tensorflow.org/api_docs/python/tf/keras/Modelpredict), passing to it a dictionary of values. The `steps` parameter determines the total number of steps before declaring the prediction round finished. Here since we have just one example, we set `steps=1` (setting `steps=None` would also work). Note, however, that if x is a `tf.data` dataset or a dataset iterator, and steps is set to None, predict will run until the input dataset is exhausted.<jupyter_code>model.predict(x={"pickup_longitude": tf.convert_to_tensor([-73.982683]),
                 "pickup_latitude": tf.convert_to_tensor([40.742104]),
                 "dropoff_longitude": tf.convert_to_tensor([-73.983766]),
                 "dropoff_latitude": tf.convert_to_tensor([40.755174]),
                 "passenger_count": tf.convert_to_tensor([3.0])},
              steps=1)<jupyter_output><empty_output><jupyter_text>Export and deploy our modelOf course, making individual predictions is not realistic, because we can't expect client code to have a model object in memory. For others to use our trained model, we'll have to export our model to a file, and expect client code to instantiate the model from that exported file. We'll export the model to a TensorFlow SavedModel format. Once we have a model in this format, we have lots of ways to "serve" the model, from a web application, from JavaScript, from mobile applications, etc. **Exercise.** Use `tf.saved_model.save` to export the trained model to a Tensorflow SavedModel format. Reference the [documentation for `tf.saved_model.save`](https://www.tensorflow.org/api_docs/python/tf/saved_model/save) as you fill in the code for the cell below.Next, print the signature of your saved model using the SavedModel Command Line Interface command `saved_model_cli`. You can read more about the command line interface and the `show` and `run` commands it supports in the [documentation here](https://www.tensorflow.org/guide/saved_modeloverview_of_commands).<jupyter_code>OUTPUT_DIR = "./export/savedmodel"
shutil.rmtree(OUTPUT_DIR, ignore_errors=True)
EXPORT_PATH = os.path.join(OUTPUT_DIR,
                           datetime.datetime.now().strftime("%Y%m%d%H%M%S"))

tf.saved_model.save(model, EXPORT_PATH) # TODO: Your code here
!saved_model_cli show \
 --tag_set serve \
--signature_def serving_default \
--dir {EXPORT_PATH} # TODO: Your code here

!find {EXPORT_PATH}
os.environ['EXPORT_PATH'] = EXPORT_PATH<jupyter_output>The given SavedModel SignatureDef contains the following input(s):
  inputs['dropoff_latitude'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, 1)
      name: serving_default_dropoff_latitude:0
  inputs['dropoff_longitude'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, 1)
      name: serving_default_dropoff_longitude:0
  inputs['passenger_count'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, 1)
      name: serving_default_passenger_count:0
  inputs['pickup_latitude'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, 1)
      name: serving_default_pickup_latitude:0
  inputs['pickup_longitude'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, 1)
      name: serving_default_pickup_longitude:0
The given SavedModel SignatureDef contains the following output(s):
  outputs['output_1'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1, 1)
      name: StatefulPartitionedCall:0
Method name is: tensorflow/serving/predict
./export/savedmodel/20201202133523
./export/save[...]<jupyter_text>Deploy our model to AI PlatformFinally, we will deploy our trained model to AI Platform and see how we can make online predicitons.  **Exercise.** Complete the code in the cell below to deploy your trained model to AI Platform using the `gcloud ai-platform versions create` command. Have a look at [the documentation for how to create model version with gcloud](https://cloud.google.com/sdk/gcloud/reference/ai-platform/versions/create).<jupyter_code>%%bash


PROJECT=$(gcloud config list --format 'value(core.project)' 2>/dev/null)
BUCKET=${PROJECT}
REGION=us-east1
MODEL_NAME=taxifare
VERSION_NAME=dnn

## Create GCS bucket if it doesn't exist already...
exists=$(gsutil ls -d | grep -w gs://${BUCKET}/)

if [ -n "$exists" ]; then
    echo -e "Bucket exists, let's not recreate it."
else
    echo "Creating a new GCS bucket."
    gsutil mb -l ${REGION} gs://${BUCKET}
    echo "\nHere are your current buckets:"
    gsutil ls
fi

if [[ $(gcloud ai-platform models list --format='value(name)' | grep $MODEL_NAME) ]]; then
    echo "$MODEL_NAME already exists"
else
    echo "Creating $MODEL_NAME"
    gcloud ai-platform models create --regions=$REGION $MODEL_NAME
fi

if [[ $(gcloud ai-platform versions list --model $MODEL_NAME --format='value(name)' | grep $VERSION_NAME) ]]; then
    echo "Deleting already existing $MODEL_NAME:$VERSION_NAME ... "
    echo yes | gcloud ai-platform versions delete --model=$MODEL_NAME $VERSION_NAME
    echo "Please run this cell again if you don't see a Creating message ... "
    sleep 2
fi

echo "Creating $MODEL_NAME:$VERSION_NAME"# TODO: Your code below
gcloud ai-platform versions create --model=$MODEL_NAME $VERSION_NAME \
    --framework=tensorflow --python-version=3.7 --runtime-version=2.1 \
    --origin=$EXPORT_PATH --staging-bucket=gs://$BUCKET
%%writefile input.json
{"pickup_longitude": -73.982683, "pickup_latitude": 40.742104,"dropoff_longitude": -73.983766,"dropoff_latitude": 40.755174,"passenger_count": 3.0}<jupyter_output>Overwriting input.json<jupyter_text>**Exercise.** Complete the code in the cell below to call prediction on your deployed model for the example you just created in the `input.json` file above.<jupyter_code>!gcloud ai-platform predict \
    --model taxifare \
--json-instances input.json \
--version dnn
# TODO: Your code above<jupyter_output>Using endpoint [https://ml.googleapis.com/]
OUTPUT_1
[11.762494087219238]
{'fraction_non_alphanumeric': 0.1099959855479727, 'fraction_numerical': 0.05570052187876355, 'mean_word_length': 4.545075125208681, 'pattern_counts': {'":': 10, '<': 52, '<?xml version=': 0, '>': 53, 'https://': 12, 'lorem ipsum': 0, 'www.': 7, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '17480', 'license': 'Apache-2.0', 'n_tokens_mistral': 6750, 'n_tokens_neox': 6026, 'n_words': 2095, 'path': 'notebooks/introduction_to_tensorflow/labs/3_keras_sequential_api.ipynb', 'repo_name': 'jfesteban/Google-ASL', 'size': '19928'}