<jupyter_start><jupyter_text>EKS CSI FSX Lustre SetupAmazon FSx for Lustre is a high-performance file system optimized for deep learning workloads. FSx provides POSIX-compliant file system access to S3 for multiple readers and writers simultaneously.  The Amazon FSx for Lustre Container Storage Interface (CSI) driver provides a CSI interface that allows Amazon EKS clusters to manage the lifecycle of Amazon FSx for Lustre file systems.  * https://docs.aws.amazon.com/eks/latest/userguide/fsx-csi.html* https://github.com/kubernetes-sigs/aws-fsx-csi-driver<jupyter_code>import boto3
import json
from botocore.exceptions import ClientError

iam = boto3.client('iam')
sts = boto3.client('sts')
cfn = boto3.client('cloudformation')
eks = boto3.client('eks')

region = boto3.Session().region_name
cluster_name = 'workshop'<jupyter_output><empty_output><jupyter_text>1. Install the FSx CSI Driver for Kubernetes  Create IAM PolicyCreate an IAM policy and service account that allows the driver to make calls to AWS APIs on your behalf.<jupyter_code>!pygmentize fsx/fsx-csi-driver.json
# !aws iam create-policy \
#     --policy-name Amazon_FSx_Lustre_CSI_Driver \
#     --policy-document file://fsx/fsx-csi-driver.json
with open('fsx/fsx-csi-driver.json') as json_file:
    data = json.load(json_file)
    policy = json.dumps(data)

try:
    response = iam.create_policy(
        PolicyName='Amazon_FSx_Lustre_CSI_Driver',
        PolicyDocument=policy
    )
    print("[OK] Policy created.")

except ClientError as e:
    if e.response['Error']['Code'] == 'EntityAlreadyExists':
        print("[OK] Policy already exists.")
    else:
        print("Error: %s" % e)
account_id = sts.get_caller_identity()['Account']
csi_policy_arn = 'arn:aws:iam::{}:policy/Amazon_FSx_Lustre_CSI_Driver'.format(account_id)
print(csi_policy_arn)<jupyter_output>arn:aws:iam::231218423789:policy/Amazon_FSx_Lustre_CSI_Driver<jupyter_text>Create Kubernetes IAM Service AccountCreate a Kubernetes service account for the driver and attach the policy to the service account. Replacing the ARN of the policy with the ARN returned in the previous step.  _The next cell runs for about 10min. Please be patient._<jupyter_code>!eksctl create iamserviceaccount \
     --region $region \
     --name fsx-csi-controller-sa \
     --namespace kube-system \
     --cluster $cluster_name \
     --attach-policy-arn $csi_policy_arn \
     --approve
cf_stack_name = 'eksctl-{}-addon-iamserviceaccount-kube-system-fsx-csi-controller-sa'.format(cluster_name)
print(cf_stack_name)
response = cfn.list_stack_resources(
    StackName=cf_stack_name
)
print(response)
iam_role_name = response['StackResourceSummaries'][0]['PhysicalResourceId']
print(iam_role_name)
iam_role_arn = iam.get_role(RoleName=iam_role_name)['Role']['Arn']
print(iam_role_arn)<jupyter_output>arn:aws:iam::231218423789:role/eksctl-workshop-addon-iamserviceaccount-kube-Role1-YQ8E1QVLFGWV<jupyter_text>Deploy CSI Driver<jupyter_code>!kubectl apply -k "github.com/kubernetes-sigs/aws-fsx-csi-driver/deploy/kubernetes/overlays/stable/?ref=master"<jupyter_output>Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
serviceaccount/fsx-csi-controller-sa configured
clusterrole.rbac.authorization.k8s.io/fsx-csi-external-provisioner-role created
clusterrolebinding.rbac.authorization.k8s.io/fsx-csi-external-provisioner-binding created
deployment.apps/fsx-csi-controller created
daemonset.apps/fsx-csi-node created
csidriver.storage.k8s.io/fsx.csi.aws.com created<jupyter_text>Patch the driver deployment to add the service account that you just created, replacing the ARN with the correct role ARN.<jupyter_code>!kubectl annotate serviceaccount -n kube-system fsx-csi-controller-sa \
 eks.amazonaws.com/role-arn=$iam_role_arn --overwrite=true<jupyter_output>serviceaccount/fsx-csi-controller-sa annotated<jupyter_text>Check S3 Bucket For FSX<jupyter_code>bucket = 's3://fsx-antje'
#!aws s3 mb $bucket
!aws s3 ls $bucket
!aws s3 ls $bucket --recursive<jupyter_output>2020-11-21 16:02:48          0 code/test_data/
2020-11-21 16:02:48   18997559 code/test_data/amazon_reviews_us_Digital_Software_v1_00.tsv.gz
2020-11-21 16:21:10      21708 code/train.py
2020-10-30 18:14:13        615 input/data/test/part-algo-1-amazon_reviews_us_Digital_Software_v1_00.tfrecord
2020-10-30 18:14:13        632 input/data/test/part-algo-2-amazon_reviews_us_Digital_Video_Games_v1_00.tfrecord
2020-10-30 18:14:13      10728 input/data/train/part-algo-1-amazon_reviews_us_Digital_Software_v1_00.tfrecord
2020-10-30 18:14:13      11812 input/data/train/part-algo-2-amazon_reviews_us_Digital_Video_Games_v1_00.tfrecord
2020-10-30 18:14:13        679 input/data/validation/part-algo-1-amazon_reviews_us_Digital_Software_v1_00.tfrecord
2020-10-30 18:14:13        642 input/data/validation/part-algo-2-amazon_reviews_us_Digital_Video_Games_v1_00.tfrecord
2020-10-30 18:14:43          0 model/<jupyter_text>Download Storage Class Manifest<jupyter_code>!curl -o storageclass.yaml https://raw.githubusercontent.com/kubernetes-sigs/aws-fsx-csi-driver/master/examples/kubernetes/dynamic_provisioning_s3/specs/storageclass.yaml<jupyter_output><empty_output><jupyter_text>Get VPC ID and Subnet ID<jupyter_code>%%bash

source ~/.bash_profile

#### Get VPC ID
export VPC_ID=$(aws ec2 describe-vpcs --filters "Name=tag:Name,Values=eksctl-${AWS_CLUSTER_NAME}-cluster/VPC" --query "Vpcs[0].VpcId" --output text)
echo "export VPC_ID=${VPC_ID}" | tee -a ~/.bash_profile

#### Get Subnet ID
export SUBNET_ID=$(aws ec2 describe-subnets --filters "Name=vpc-id,Values=${VPC_ID}" --query "Subnets[0].SubnetId" --output text)
echo "export SUBNET_ID=${SUBNET_ID}" | tee -a ~/.bash_profile<jupyter_output>export VPC_ID=vpc-02e8cba0a081bf4ad
export SUBNET_ID=subnet-0a25103d821733ac0<jupyter_text>Create Security Group<jupyter_code>%%bash

source ~/.bash_profile

export SEC_GROUP_ID=$(aws ec2 create-security-group --group-name eks-fsx-security-group --vpc-id ${VPC_ID} --description "FSx for Lustre Security Group" --query "GroupId" --output text)
echo "export SEC_GROUP_ID=${SEC_GROUP_ID}" | tee -a ~/.bash_profile<jupyter_output>export SEC_GROUP_ID=sg-0ee7f41b8573a5d2b<jupyter_text>Add an ingress rule that opens up port 988 from the 192.168.0.0/16 CIDR range<jupyter_code>%%bash

source ~/.bash_profile

aws ec2 authorize-security-group-ingress --group-id ${SEC_GROUP_ID} --protocol tcp --port 988 --cidr 192.168.0.0/16<jupyter_output><empty_output><jupyter_text>Update the environment variables in the `storageclass.yaml` file<jupyter_code>!pygmentize fsx/storageclass.yaml<jupyter_output>[94mkind[39;49;00m: StorageClass
[94mapiVersion[39;49;00m: storage.k8s.io/v1
[94mmetadata[39;49;00m:
  [94mname[39;49;00m: fsx-sc
[94mprovisioner[39;49;00m: fsx.csi.aws.com
[94mparameters[39;49;00m:
  [94msubnetId[39;49;00m: subnet-0a25103d821733ac0
  [94msecurityGroupIds[39;49;00m: sg-0ee7f41b8573a5d2b
  [94ms3ImportPath[39;49;00m: s3://fsx-antje
  [94ms3ExportPath[39;49;00m: s3://fsx-antje
  [94mautoImportPolicy[39;49;00m: NEW_CHANGED
  [94mdeploymentType[39;49;00m: SCRATCH_2
[94mmountOptions[39;49;00m:
  - flock<jupyter_text>Create FSX Storage Class<jupyter_code>#!kubectl delete -f fsx/storageclass.yaml
!kubectl create -f fsx/storageclass.yaml
!kubectl get sc<jupyter_output>NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
fsx-sc          fsx.csi.aws.com         Delete          Immediate              false                  3s
gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  36m<jupyter_text>Create Claim<jupyter_code>#!curl -o claim.yaml https://raw.githubusercontent.com/kubernetes-sigs/aws-fsx-csi-driver/master/examples/kubernetes/dynamic_provisioning_s3/specs/claim.yaml
!pygmentize fsx/claim.yaml
#!kubectl delete -f fsx/claim.yaml
!kubectl apply -f fsx/claim.yaml
!kubectl get pvc fsx-claim
!kubectl describe pvc fsx-claim<jupyter_output>Name:          fsx-claim
Namespace:     default
StorageClass:  fsx-sc
Status:        Bound
Volume:        pvc-a7b6809e-e622-45d6-a754-d8b9d0ee6e25
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: fsx.csi.aws.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      1200Gi
Access Modes:  RWX
VolumeMode:    Filesystem
Mounted By:    <none>
Events:
  Type     Reason                Age                 From                                                                                      Message
  ----     ------                ----                ----                                                                                      -------
  Normal   Provisioning          11m (x3 over 21m)   fsx.csi.aws.com_fsx-csi-controller-55bcb55d5d-stdbn_34284e3c-2c25-11eb-8987-228ffe6b94e2  External provisioner is provisioning volume for claim "d[...]<jupyter_text>_Wait for status == Bound_  Update FSX to `autoImportPolicy: NEW_CHANGED`<jupyter_code>fsx = boto3.client('fsx')
response = fsx.describe_file_systems()
fsx_id = response['FileSystems'][0]['FileSystemId']
print(fsx_id)
response = fsx.update_file_system(
    FileSystemId=fsx_id,
    LustreConfiguration={
        'AutoImportPolicy': 'NEW_CHANGED'
    }
)
print(response)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11235836069045854, 'fraction_numerical': 0.054114158636026685, 'mean_word_length': 4.18331503841932, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '439954', 'license': 'Apache-2.0', 'n_tokens_mistral': 3778, 'n_tokens_neox': 3395, 'n_words': 608, 'path': '10_pipeline/kubeflow/wip/dlc/00_setup_eks/00_04_Setup_FSX.ipynb', 'repo_name': 'MarcusFra/workshop', 'size': '9443'}