<jupyter_start><jupyter_text>Kubeflow Fairing Kaniko Cloud Builder on AWS Requirements  * You must be running Kubeflow 1.0 or newer on EKS  Create AWS secret in kubernetes and grant aws access to your notebook> Note: Once IAM for Service Account is merged in 1.0.1, we don't have to use credentials1. Please create an AWS secret in current namespace. > Note: To get base64 string, try `echo -n $AWS_ACCESS_KEY_ID | base64`. > Make sure you have `AmazonEC2ContainerRegistryFullAccess` and `AmazonS3FullAccess` for this experiment. Pods will use credentials to talk to AWS services.<jupyter_code>%%bash

# Replace placeholder with your own AWS credentials
AWS_ACCESS_KEY_ID='<YOUR AWS ACCESS KEY>'
AWS_SECRET_ACCESS_KEY='<YOUR AWS SECRET KEY>'

kubectl create secret generic aws-secret --from-literal=AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} --from-literal=AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}<jupyter_output><empty_output><jupyter_text>2. Attach `AmazonEC2ContainerRegistryFullAccess` and `AmazonS3FullAccess` to EKS node group role and grant AWS access to notebook.  Verify you have access to AWS services* The cell below checks that this notebook was spawned with credentials to access AWS S3 and ECR<jupyter_code>import logging
import os
import uuid
from importlib import reload
import boto3

# Set REGION for s3 bucket and elastic contaienr registry
AWS_REGION='us-west-2'
boto3.client('s3', region_name=AWS_REGION).list_buckets()
boto3.client('ecr', region_name=AWS_REGION).describe_repositories()<jupyter_output><empty_output><jupyter_text>Install Required LibrariesImport the libraries required to train this model.<jupyter_code>from src.kaniko import notebook_setup
reload(notebook_setup)
notebook_setup.notebook_setup()

# Force a reload of kubeflow; since kubeflow is a multi namespace module
# it looks like doing this in notebook_setup may not be sufficient
import kubeflow
reload(kubeflow)<jupyter_output><empty_output><jupyter_text>Configure ECR Docker Registry For Kubeflow Fairing* In order to build docker images from your notebook we need a docker registry where the images will be stored* Below you set some variables specifying a [Amazon Elastic Container Registry](https://aws.amazon.com/ecr/)* Kubeflow Fairing provides a utility function to guess the name of your AWS account<jupyter_code>from kubeflow import fairing   
from kubeflow.fairing import utils as fairing_utils
from kubeflow.fairing.preprocessors import base as base_preprocessor
from kubeflow.tfjob.api import tf_job_client as tf_job_client_module
import yaml

# Setting up AWS Elastic Container Registry (ECR) for storing output containers
# You can use any docker container registry istead of ECR
AWS_ACCOUNT_ID=fairing.cloud.aws.guess_account_id()
AWS_ACCOUNT_ID = boto3.client('sts').get_caller_identity().get('Account')
DOCKER_REGISTRY = '{}.dkr.ecr.{}.amazonaws.com'.format(AWS_ACCOUNT_ID, AWS_REGION)

namespace = fairing_utils.get_current_k8s_namespace()

logging.info(f"Running in aws region {AWS_REGION}, account {AWS_ACCOUNT_ID}")
logging.info(f"Running in namespace {namespace}")
logging.info(f"Using docker registry {DOCKER_REGISTRY}")<jupyter_output><empty_output><jupyter_text>Use Kubeflow fairing to build the docker image* You will use kubeflow fairing's kaniko builder to build a docker image that includes all your dependencies  * You use kaniko because you want to be able to run `pip` to install dependencies  * Kaniko gives you the flexibility to build images from Dockerfiles<jupyter_code>from kubeflow.fairing.builders import cluster

# output_map is a map of extra files to add to the notebook.
# It is a map from source location to the location inside the context.
output_map =  {
    "./src/kaniko/Dockerfile.model": "Dockerfile",
    "./src/kaniko/model.py": "model.py"
}

preprocessor = base_preprocessor.BasePreProcessor(
    command=["python"], # The base class will set this.
    input_files=[],
    path_prefix="/app", # irrelevant since we aren't preprocessing any files
    output_map=output_map)

preprocessor.preprocess()
# Create a new ECR repository to host model image
ecr_repo_name = 'fairing-kaniko'
!aws ecr create-repository --repository-name $ecr_repo_name --region=$AWS_REGION
# Use a Tensorflow image as the base image
# We use a custom Dockerfile 
cluster_builder = cluster.cluster.ClusterBuilder(registry=DOCKER_REGISTRY,
                                                 base_image="", # base_image is set in the Dockerfile
                                                 preprocessor=preprocessor,
                                                 image_name="fairing-kaniko",
                                                 dockerfile_path="Dockerfile",
                                                 pod_spec_mutators=[fairing.cloud.aws.add_aws_credentials_if_exists, fairing.cloud.aws.add_ecr_config],
                                                 context_source=cluster.s3_context.S3ContextSource(region=AWS_REGION))
cluster_builder.build()
logging.info(f"Built image {cluster_builder.image_tag}")<jupyter_output><empty_output><jupyter_text>Create a S3 Bucket* Create a S3 bucket to store our models and other results.* Since we are running in python we use the python client libraries.<jupyter_code>import boto3
from botocore.exceptions import ClientError

bucket = f"{AWS_ACCOUNT_ID}-fairing-kaniko"

def create_bucket(bucket_name):
    """Create an S3 bucket in a specified region

    :param bucket_name: Bucket to create
    :return: True if bucket created, else False
    """

    # Create bucket
    try:
        s3_client = boto3.client('s3')
        s3_client.create_bucket(Bucket=bucket_name)
    except ClientError as e:
        logging.error(e)
        return False
    return True

create_bucket(bucket)<jupyter_output><empty_output><jupyter_text>Distributed training by using Fairing Kaniko image* We will train the model by using TFJob to run a distributed training job<jupyter_code>from src.kaniko.tfjob_spec_provider import tfj_spec 
train_name=f"mnist-train-{uuid.uuid4().hex[:4]}"
train_spec = tfj_spec(train_name=train_name,
                    num_ps=1,
                    num_workers=2,
                    model_dir=f"s3://{bucket}/mnist",
                    export_path=f"s3://{bucket}/mnist/export",
                    train_steps=200,
                    batch_size=100,
                    learning_rate=.01,
                    image=cluster_builder.image_tag,
                    AWS_REGION=AWS_REGION)<jupyter_output><empty_output><jupyter_text>Create the training job from Kaniko Image* You could write the spec to a YAML file and then do `kubectl apply -f {FILE}`* Since you are running in jupyter you will use the TFJob client* You will run the TFJob in a namespace created by a Kubeflow profile  * The namespace will be the same namespace you are running the notebook in  * Creating a profile ensures the namespace is provisioned with service accounts and other resources needed for Kubeflow<jupyter_code>tf_job_client = tf_job_client_module.TFJobClient()
tf_job_body = yaml.safe_load(train_spec)
tf_job = tf_job_client.create(tf_job_body, namespace=namespace)  

logging.info(f"Created job {namespace}.{train_name}")<jupyter_output><empty_output><jupyter_text>Check the job* Above you used the python SDK for TFJob to check the status* You can also use kubectl get the status of your job* The job conditions will tell you whether the job is running, succeeded or failed<jupyter_code>!kubectl get tfjobs -o yaml {train_name}<jupyter_output><empty_output><jupyter_text>Clean Up<jupyter_code>!aws ecr delete-repository --repository-name $ecr_repo_name --force --region=$AWS_REGION
!aws s3 rb s3://$bucket --force<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07309184993531695, 'fraction_numerical': 0.00646830530401035, 'mean_word_length': 4.174698795180723, 'pattern_counts': {'":': 2, '<': 43, '<?xml version=': 0, '>': 46, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '387555', 'license': 'Apache-2.0', 'n_tokens_mistral': 2341, 'n_tokens_neox': 2279, 'n_words': 726, 'path': 'notebooks/02_Fairing/02_04_fairing_kaniko_cloud_builder.ipynb', 'repo_name': 'bowzoo/eks-kubeflow-workshop', 'size': '7730'}