<jupyter_start><jupyter_text>Set gcloud project<jupyter_code>#!gcloud config set project YOUR_PROJECT_ID
project_id = !gcloud config list --format 'value(core.project)' 2>/dev/null
print(project_id)
#!gcloud config list<jupyter_output><empty_output><jupyter_text>Environmental variables<jupyter_code>import os
os.environ['CLUSTER'] = "training-cluster"
os.environ['PROJECT_ID'] = project_id[0]<jupyter_output><empty_output><jupyter_text>Pre-Requisites to Setup GKE Cluster  1. gcloud command - Create cluster<jupyter_code>!gcloud container clusters create training-cluster-beta \
--num-nodes=2 \
--zone=us-central1-b \
--workload-pool=${PROJECT_ID}.svc.id.goog<jupyter_output><empty_output><jupyter_text>2. Create Kubernetes service account<jupyter_code>!kubectl create serviceaccount sa-trainer<jupyter_output><empty_output><jupyter_text>3. Create Google service account<jupyter_code>gcloud iam service-accounts create gke-trainer-sa<jupyter_output><empty_output><jupyter_text>4. Create IAM policy binding that allows Kubernetes SA to impersonate the Google service account. This binding allows the Kubernetes Service account to act as the Google service account.<jupyter_code>!gcloud iam service-accounts add-iam-policy-binding \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:virtual-anomaly.svc.id.goog[default/sa-trainer]" \
<EMAIL><jupyter_output><empty_output><jupyter_text>5. Add storage role to the Google service account<jupyter_code>!gcloud projects add-iam-policy-binding virtual-anomaly \
--member "serviceAccount:<EMAIL>" \
--role "role/storage.objectAdmin"<jupyter_output><empty_output><jupyter_text>6. Connect to the cluster (Command can also be found in Cloud Console for GKE) Configure kubectl  command line access by running the following command:<jupyter_code>!gcloud container clusters get-credentials training-cluster-beta --zone us-central1-b --project $PROJECT_ID<jupyter_output><empty_output><jupyter_text>7. Add annotation to the Kubernetes service account, using the email address of the Google service account.<jupyter_code>!kubectl annotate serviceaccount \
--namespace default sa-trainer \
iam.gke.io/gcp-service-account=<EMAIL><jupyter_output><empty_output><jupyter_text>Refer to https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity for the latest instructions.  Run Training and Prediction Jobs on GKE Cluster  Run the Kubernetes training job<jupyter_code>!kubectl apply -f k8s_job_training.yaml<jupyter_output><empty_output><jupyter_text>VerifyVerify that the output cloud storage location specified in the yaml has the contents with the timestamp matching the job run time.  Run the Kubernetes prediction job<jupyter_code>!kubectl apply -f k8s_job_prediction.yaml<jupyter_output><empty_output><jupyter_text>VerifyVerify that the output cloud storage location specified in the yaml has the contents with the timestamp matching the job run time.  Troubleshooting 101 kubectl get jobs  Lists all jobs in the default namespacekubectl get pods  Lists all pods in the default namespacekubectl describe job {pod_name}  Gives additional information about the jobkubectl describe pod {pod_name}  Gives additional information about the podkubectl logs {pod_name}  Shows the pod logs<jupyter_code>!kubectl get jobs<jupyter_output><empty_output><jupyter_text>Rerun Training<jupyter_code>!kubectl delete job trainer-job<jupyter_output><empty_output><jupyter_text>Rerun Prediction/Infererence<jupyter_code>!kubectl delete job prediction-job<jupyter_output><empty_output><jupyter_text>Destroy the cluster<jupyter_code>!gcloud container clusters delete training-cluster<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08481532147742818, 'fraction_numerical': 0.004651162790697674, 'mean_word_length': 7.462962962962963, 'pattern_counts': {'":': 0, '<': 64, '<?xml version=': 0, '>': 65, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '217166', 'license': 'Apache-2.0', 'n_tokens_mistral': 1184, 'n_tokens_neox': 1158, 'n_words': 293, 'path': 'sklearn-custom-container-gke/3. gke-trainer-prediction.ipynb', 'repo_name': 'malini-kamalambal/vais', 'size': '3796'}