<jupyter_start><jupyter_text>Distributed Training**Learning Objectives**  - Use AI Platform Training Service to run a distributed training job IntroductionIn the previous notebook we trained our model on AI Platform Training Service, but we didn't recieve any benefit. In fact it was much slower to train on the Cloud (5-10 minutes) than it was to train locally! Why is this?**1. The job was too small**AI Platform Training Service provisions hardware on-demand. This is good because it means you only pay for what you use, but for small jobs it means the start up time for the hardware is longer than the training time itself!To address this we'll use a dataset that is 100x as big, and enough steps to go through all the data at least once.**2. The hardware was too small**By default AI Platform Training Service jobs train on an [n1-standard-4](https://cloud.google.com/compute/docs/machine-typesstandard_machine_types) instance, which isn't that much more powerful than our local VM. And even if it was we could [easily increase the specs](https://cloud.google.com/compute/docs/instances/changing-machine-type-of-stopped-instance) of our local VM to match.To get the most benefit out of AI Platform Training Service we need to move beyond training on a single instance and instead train across multiple machines.Because we're using `tf.estimator.train_and_evaluate()`, our model already knows how to distribute itself while training! So all we need to do is supply a `--scale-tier` parameter to the AI Platform Training Service train job which will provide the distributed training environment. See the different scale tiers avaialable [here](https://cloud.google.com/ml-engine/docs/tensorflow/machine-typesscale_tiers). We will use STANDARD_1 which corresponds to  1 n1-highcpu-8 master instance, 4 n1-highcpu-8 worker instances, and n1-standard-4 3 parameter servers. We will cover the details of the distribution strategy and why there are master/worker/parameter designations later in the course. Training will take about 20 minutes<jupyter_code>PROJECT = "qwiklabs-gcp-636667ae83e902b6"  # Replace with your PROJECT
BUCKET = "qwiklabs-gcp-636667ae83e902b6_al"  # Replace with your BUCKET
REGION = "us-east1"            # Choose an available region for AI Platform Training Service
TFVERSION = "1.13"                # TF version for AI Platform Training Service to use<jupyter_output><empty_output><jupyter_text>Run distributed cloud jobAfter having testing our training pipeline both locally and in the cloud on a susbset of the data, we'll now submit another (much larger) training job to the cloud. The `gcloud` command is almost exactly the same though we'll need to alter some of the previous parameters to point our training job at the much larger dataset. You can follow these links to read more about ["Using Distributed TensorFlow with Cloud ML Engine"](https://cloud.google.com/ml-engine/docs/tensorflow/distributed-tensorflow-mnist-cloud-datalab) or ["Specifying Machine Types or Scale Tiers"](https://cloud.google.com/ml-engine/docs/tensorflow/machine-typesscale_tiers). Note the `train_data_path` and `eval_data_path` in the Exercise code below as well `train_steps`, the number of training steps.To start, we'll set up our output directory as before, now calling it `trained_large`. Then we submit the training job using `gcloud ai-platform` similar to before.<jupyter_code>OUTDIR = "gs://{}/taxifare/trained_large".format(BUCKET)
!gsutil -m rm -rf {OUTDIR} # start fresh each time
!gcloud ai-platform jobs submit training taxifare_large_$(date -u +%y%m%d_%H%M%S) \
    --package-path=taxifaremodel \
    --module-name=taxifaremodel.task \
    --job-dir=gs://{BUCKET}/taxifare \
    --python-version=3.5 \
    --runtime-version={TFVERSION} \
    --region={REGION} \
    --scale-tier=STANDARD_1 \
    -- \
    --train_data_path=gs://cloud-training-demos/taxifare/large/taxi-train*.csv \
    --eval_data_path=gs://cloud-training-demos/taxifare/small/taxi-valid.csv  \
    --train_steps=200000 \
    --output_dir={OUTDIR}<jupyter_output>Removing gs://qwiklabs-gcp-636667ae83e902b6_al/taxifare/trained_large/#1563392722981251...
Removing gs://qwiklabs-gcp-636667ae83e902b6_al/taxifare/trained_large/checkpoint#1563392726107134...
Removing gs://qwiklabs-gcp-636667ae83e902b6_al/taxifare/trained_large/events.out.tfevents.1563392691.cmle-training-master-ba8091646c-0-pdq5k#1563392691936105...
Removing gs://qwiklabs-gcp-636667ae83e902b6_al/taxifare/trained_large/events.out.tfevents.1563392694.cmle-training-worker-ba8091646c-0-r6gnm#1563392694791992...
Removing gs://qwiklabs-gcp-636667ae83e902b6_al/taxifare/trained_large/model.ckpt-0.data-00000-of-00004#1563392724891508...
Removing gs://qwiklabs-gcp-636667ae83e902b6_al/taxifare/trained_large/model.ckpt-0.data-00001-of-00004#1563392724367150...
Removing gs://qwiklabs-gcp-636667ae83e902b6_al/taxifare/trained_large/model.ckpt-0.meta#1563392726685788...
Removing gs://qwiklabs-gcp-636667ae83e902b6_al/taxifare/trained_large/graph.pbtxt#1563392721190574...
Removing gs://qwiklabs-gcp-636[...]
{'fraction_non_alphanumeric': 0.0936573799644339, 'fraction_numerical': 0.06856352499506027, 'mean_word_length': 4.745743473325766, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '80437', 'license': 'Apache-2.0', 'n_tokens_mistral': 1785, 'n_tokens_neox': 1575, 'n_words': 486, 'path': 'courses/machine_learning/deepdive/02_tensorflow/g_distributed.ipynb', 'repo_name': 'aleistra/training-data-analyst', 'size': '5061'}