<jupyter_start><jupyter_text>Copyright 2021 The TensorFlow Cloud Authors.<jupyter_code>#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.<jupyter_output><empty_output><jupyter_text>Distributed training NasNet with tensorflow_cloud and Google Cloud      View on TensorFlow.org        Run in Google Colab        View on GitHub        Download notebook                        Run in Kaggle              This example is based on [Image classification via fine-tuning with EfficientNet](https://keras.io/examples/vision/image_classification_efficientnet_fine_tuning/) to demonstrate how to train a [NasNetMobile](https://keras.io/api/applications/nasnet/nasnetmobile-function) model using [tensorflow_cloud](https://tensorflow.org/cloud) and Google Cloud Platform at scale using distributed training.  Import required modules<jupyter_code>import tensorflow as tf
tf.version.VERSION
! pip install -q tensorflow-cloud

import tensorflow_cloud as tfc
tfc.__version__
import sys
import os<jupyter_output><empty_output><jupyter_text>Project ConfigurationsSet project parameters. For Google Cloud Specific parameters refer to [Google Cloud Project Setup Instructions](https://www.kaggle.com/nitric/google-cloud-project-setup-instructions/).<jupyter_code># Set Google Cloud Specific parameters

# TODO: Please set GCP_PROJECT_ID to your own Google Cloud project ID.
GCP_PROJECT_ID = 'YOUR_PROJECT_ID' #@param {type:"string"}

# TODO: set GCS_BUCKET to your own Google Cloud Storage (GCS) bucket.
GCS_BUCKET = 'YOUR_GCS_BUCKET_NAME' #@param {type:"string"}

# DO NOT CHANGE: Currently only the 'us-central1' region is supported.
REGION = 'us-central1'

# OPTIONAL: You can change the job name to any string.
JOB_NAME = 'nasnet' #@param {type:"string"}

# Setting location were training logs and checkpoints will be stored
GCS_BASE_PATH = f'gs://{GCS_BUCKET}/{JOB_NAME}'
TENSORBOARD_LOGS_DIR = os.path.join(GCS_BASE_PATH,"logs")
MODEL_CHECKPOINT_DIR = os.path.join(GCS_BASE_PATH,"checkpoints")
SAVED_MODEL_DIR = os.path.join(GCS_BASE_PATH,"saved_model")<jupyter_output><empty_output><jupyter_text>Authenticating the notebook to use your Google Cloud ProjectFor Kaggle Notebooks click on "Add-ons"->"Google Cloud SDK" before running the cell below.<jupyter_code># Using tfc.remote() to ensure this code only runs in notebook
if not tfc.remote():

    # Authentication for Kaggle Notebooks
    if "kaggle_secrets" in sys.modules:
        from kaggle_secrets import UserSecretsClient
        UserSecretsClient().set_gcloud_credentials(project=GCP_PROJECT_ID)

    # Authentication for Colab Notebooks
    if "google.colab" in sys.modules:
        from google.colab import auth
        auth.authenticate_user()
        os.environ["GOOGLE_CLOUD_PROJECT"] = GCP_PROJECT_ID<jupyter_output><empty_output><jupyter_text>Load and prepare dataRead raw data and split to train and test data sets.<jupyter_code>(x_train, y_train), (x_test, y_test) = tf.keras.datasets.cifar10.load_data()

# Setting input specific parameters
# The model expects input of dimension (INPUT_IMG_SIZE, INPUT_IMG_SIZE, 3)
INPUT_IMG_SIZE = 32
NUM_CLASSES = 10<jupyter_output><empty_output><jupyter_text>Add preprocessing layers APIs for image augmentation.<jupyter_code>from tensorflow.keras.layers.experimental import preprocessing
from tensorflow.keras.models import Sequential


img_augmentation = Sequential(
    [
        # Resizing input to better match ImageNet size
        preprocessing.Resizing(256, 256),
        preprocessing.RandomRotation(factor=0.15),
        preprocessing.RandomFlip(),
        preprocessing.RandomContrast(factor=0.1),
    ],
    name="img_augmentation",
)<jupyter_output><empty_output><jupyter_text>Load the model and prepare for trainingWe will load a NASNetMobile pretrained model (with weights) and unfreeze a few layers for fine tuning the model to better match the dataset.<jupyter_code>from tensorflow.keras import layers

def build_model(num_classes, input_image_size):
    inputs = layers.Input(shape=(input_image_size, input_image_size, 3))
    x = img_augmentation(inputs)

    model = tf.keras.applications.NASNetMobile(
        input_shape=None,
        include_top=False,
        weights="imagenet",
        input_tensor=x,
        pooling=None,
        classes=num_classes,
    )

    # Freeze the pretrained weights
    model.trainable = False

    # Unfreeze the top 20 layers while leaving BatchNorm layers frozen
    for layer in model.layers[-20:]:
        if not isinstance(layer, layers.BatchNormalization):
            layer.trainable = True

    # Rebuild non-frozen top layers of NASNetMobile, which was initialized with include_top=False
    x = layers.GlobalAveragePooling2D(name="avg_pool")(model.output)
    x = layers.BatchNormalization()(x)

    x = layers.Dense(128, activation="relu")(x)
    x = layers.Dense(64, activation="relu")(x)
    outputs = layers.Dense(num_classes, activation="softmax", name="pred")(x)

    # Compile
    model = tf.keras.Model(inputs, outputs, name="NASNetMobile")
    optimizer = tf.keras.optimizers.Adam(learning_rate=3e-4)
    model.compile(
        optimizer=optimizer,
        loss="sparse_categorical_crossentropy",
        metrics=["accuracy"]
    )
    return model
model = build_model(NUM_CLASSES, INPUT_IMG_SIZE)

# Set up job on TensorFlow Cloud to train for 100 epochs on all the data
if tfc.remote():
    # Configure Tensorboard logs
    callbacks=[
        tf.keras.callbacks.TensorBoard(log_dir=TENSORBOARD_LOGS_DIR),
        tf.keras.callbacks.ModelCheckpoint(
            MODEL_CHECKPOINT_DIR,
            save_best_only=True),
        tf.keras.callbacks.EarlyStopping(
            monitor='loss',
            min_delta =0.001,
            patience=3)]

    model.fit(x=x_train, y=y_train, epochs=100,
              validation_split=0.2, callbacks=callbacks)

    model.save(SAVED_MODEL_DIR)

else:
    # Run the training locally for 1 epoch and a small subset of the data to validate setup
    model.fit(x=x_train[:100], y=y_train[:100], validation_split=0.2, epochs=1)<jupyter_output><empty_output><jupyter_text>Start the remote trainingThis step will prepare your code from this notebook for remote execution and starts a distributed training remotely on Google Cloud Platform to train the model. Once the job is submitted you can go to the next step to monitor the jobs progress via Tensorboard.<jupyter_code># You can install custom modules for your Docker images via requirements txt file.
with open('requirements.txt','w') as f:
    f.write('tensorflow-cloud\n')

# Optional: Some recommended base images. If you provide none the system
# will choose one for you.
TF_GPU_IMAGE= "gcr.io/deeplearning-platform-release/tf2-cpu.2-5"
TF_CPU_IMAGE= "gcr.io/deeplearning-platform-release/tf2-gpu.2-5"

# Submit a distributed training job using GPUs.
tfc.run(
    distribution_strategy='auto',
    requirements_txt='requirements.txt',
    docker_config=tfc.DockerConfig(
        parent_image=TF_GPU_IMAGE,
        image_build_bucket=GCS_BUCKET
        ),
    chief_config=tfc.COMMON_MACHINE_CONFIGS['K80_1X'],
      worker_config=tfc.COMMON_MACHINE_CONFIGS['K80_1X'],
      worker_count=3,
    job_labels={'job': JOB_NAME}
)<jupyter_output><empty_output><jupyter_text>Training Results Reconnect your Colab instanceMost remote training jobs are long running. If you are using Colab, it may time out before the training results are available. In that case, rerun the following sections to reconnect and configure your Colab instance to access the training results. Run the following sections in order:1.   Import required modules2.   Project Configurations3.   Authenticating the notebook to use your Google Cloud Project Load TensorboardWhile the training is in progress you can use Tensorboard to view the results. Note the results will show only after your training has started. This may take a few minutes.<jupyter_code>%load_ext tensorboard
%tensorboard --logdir $TENSORBOARD_LOGS_DIR<jupyter_output><empty_output><jupyter_text>Load your trained model<jupyter_code>trained_model = tf.keras.models.load_model(SAVED_MODEL_DIR)
trained_model.summary()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08208868820192196, 'fraction_numerical': 0.009262475396549728, 'mean_word_length': 4.537179487179487, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 42, 'https://': 5, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '145560', 'license': 'Apache-2.0', 'n_tokens_mistral': 2670, 'n_tokens_neox': 2588, 'n_words': 758, 'path': 'g3doc/tutorials/distributed_training_nasnet_with_tensorflow_cloud.ipynb', 'repo_name': 'anukaal/cloud', 'size': '8637'}