<jupyter_start><jupyter_text>Evaluating a Model with Dataflow and BigQuery This notebook is the third in the set of steps to run machine learning on the cloud. In this step, we will use the model training in the [previous notebook](./3 Service Train.ipynb) and continue with evaluating the resulting model.Evaluation is accomplished by first running batch prediction over one or more evaluation datasets. This is done via Cloud Dataflow, and then analyzing the results using BigQuery. Doing evaluation using these services allow you to scale to large evaluation datasets.  Workspace SetupThe first step is to setup the workspace that we will use within this notebook - the python libraries, and the Google Cloud Storage bucket that will be used to contain the inputs and outputs produced over the course of the steps.<jupyter_code>import google.datalab as datalab
import google.datalab.ml as ml
import mltoolbox.regression.dnn as regression
import os<jupyter_output><empty_output><jupyter_text>The storage bucket was created earlier. We'll re-declare it here, so we can use it.<jupyter_code>storage_bucket = 'gs://' + datalab.Context.default().project_id + '-datalab-workspace/'
storage_region = 'us-central1'

workspace_path = os.path.join(storage_bucket, 'census')
training_path = os.path.join(workspace_path, 'training')<jupyter_output><empty_output><jupyter_text>ModelLets take a quick look at the model that was previously produced as a result of the training job.<jupyter_code>!gsutil ls -r {training_path}/model<jupyter_output>gs://cloud-ml-users-datalab-workspace/census/training/model/:
gs://cloud-ml-users-datalab-workspace/census/training/model/
gs://cloud-ml-users-datalab-workspace/census/training/model/saved_model.pb

gs://cloud-ml-users-datalab-workspace/census/training/model/assets.extra/:
gs://cloud-ml-users-datalab-workspace/census/training/model/assets.extra/
gs://cloud-ml-users-datalab-workspace/census/training/model/assets.extra/features.json
gs://cloud-ml-users-datalab-workspace/census/training/model/assets.extra/schema.json

gs://cloud-ml-users-datalab-workspace/census/training/model/variables/:
gs://cloud-ml-users-datalab-workspace/census/training/model/variables/
gs://cloud-ml-users-datalab-workspace/census/training/model/variables/variables.data-00000-of-00001
gs://cloud-ml-users-datalab-workspace/census/training/model/variables/variables.index<jupyter_text>Batch PredictionWe'll submit a batch prediction Dataflow job, to use this model by loading it into TensorFlow, and running it in `evaluation` mode (the mode that expects the input data to contain a value for target). The other mode, `prediction`, is used to predict over data where the target column is missing.**NOTE**: Batch prediction can take a few minutes to launch while compute resources are provisioned. In the case of large datasets in real-world problems, this overhead is a much smaller part of the overall job lifetime.<jupyter_code>eval_data_path = os.path.join(workspace_path, 'data/eval.csv')
evaluation_path = os.path.join(workspace_path, 'evaluation')

regression.batch_predict(training_dir=training_path, prediction_input_file=eval_data_path,
                         output_dir=evaluation_path,
                         mode='evaluation',
                         output_format='csv',
                         cloud=True)<jupyter_output>Building package and uploading to gs://cloud-ml-users-datalab-workspace/census/evaluation/staging/trainer.tar.gz
Dataflow Job submitted, see Job mltoolbox-batch-prediction-20170307193953 at https://console.developers.google.com/dataflow?project=cloud-ml-users<jupyter_text>Once prediction is done, the individual predictions will be written out into Cloud Storage.<jupyter_code>!gsutil ls {evaluation_path}
!gsutil cat {evaluation_path}/csv_schema.json
!gsutil -q -m cp -r {evaluation_path}/ /tmp
!head -n 5 /tmp/evaluation/predictions-00000*<jupyter_output>[
  {
    "type": "STRING", 
    "mode": "NULLABLE", 
    "name": "SERIALNO"
  }, 
  {
    "type": "FLOAT", 
    "mode": "NULLABLE", 
    "name": "predicted"
  }, 
  {
    "type": "FLOAT", 
    "mode": "NULLABLE", 
    "name": "target"
  }
]
1225,17.0009841919,18.0
1438,33.0704154968,45.0
2536,122.506439209,62.0
2892,12.806552887,16.0
4767,34.3795928955,58.0<jupyter_text>Analysis with BigQueryWe're going to use BigQuery to do evaluation. BigQuery can directly work against CSV data in Cloud Storage. However, if you have very large evaluation results, or you're going to be running multiple queries, it is advisable to first load the results into a BigQuery table.<jupyter_code>%bq datasource --name eval_results --paths gs://cloud-ml-users-datalab-workspace/census/evaluation/predictions*
{
  "schema": [
    {
      "type": "STRING", 
      "mode": "nullable", 
      "name": "SERIALNO"
    }, 
    {
      "type": "FLOAT", 
      "mode": "nullable", 
      "name": "predicted_target"
    }, 
    {
      "type": "FLOAT", 
      "mode": "nullable", 
      "name": "target_from_input"
    }
  ]
}
%bq query --datasource eval_results
SELECT SQRT(AVG(error)) AS rmse FROM (
  SELECT POW(target_from_input - predicted_target, 2) AS error
  FROM eval_results
)
%bq query --name distribution_query --datasource eval_results
WITH errors AS (
  SELECT predicted_target - target_from_input AS error
  FROM eval_results
),
error_stats AS (
  SELECT MIN(error) AS min_error, MAX(error) - MIN(error) AS error_range FROM errors
),
quantized_errors AS (
  SELECT error, FLOOR((error - min_error) * 20 / error_range) error_bin FROM errors CROSS JOIN error_stats
)
SELECT AVG(error) AS error, COUNT(error_bin) AS instances
FROM quantized_errors
GROUP BY error_bin
ORDER BY error_bin
%chart columns --data distribution_query --fields error,instances<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10295898944454057, 'fraction_numerical': 0.022322201072849972, 'mean_word_length': 4.427230046948357, 'pattern_counts': {'":': 19, '<': 22, '<?xml version=': 0, '>': 22, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '395566', 'license': 'Apache-2.0', 'n_tokens_mistral': 1888, 'n_tokens_neox': 1771, 'n_words': 509, 'path': 'docs/samples/ML Toolbox/Regression/Census/4 Service Evaluate.ipynb', 'repo_name': 'JeffersonH44/datalab-r', 'size': '5779'}