<jupyter_start><jupyter_text>Part 1 : Data Preparation, Process, and Store Features  [Overview](./0-AutoClaimFraudDetection.ipynb)* 0: Overview, Architecture and Data Exploration* **[1: Data Preparation, Process, and Store Features](./1-data-prep-e2e.ipynb)**  * **[Architecture](arch)**  * **[Getting started](aud-getting-started)*** 2: Train, Check Bias, Tune, Record Lineage, and Register a Model* 3: Mitigate Bias, Train New Model, Store in Registry)* 4: Deploy Model, Run Predictions* 5: Create and Run an End-to-End Pipeline to Deploy the Model The purpose of this notebook is to perform the Data Prep phase of the ML life cycle. The main Data Wrangling, data ingestion, and multiple transformations will be done through the SageMaker Studio Data Wrangler GUI.In this notebook, we will take the `.flow` files that define the transformations to the raw data. and apply them using a SageMaker Processing job that will apply those transformations to the raw data deposited in the S3 bucket as `.csv` files.   Architecture for Data Prep, Process and Store Features[overview](all-up-overview)----![Data Prep and Store](./images/wrangler.png)  Install required and/or update third-party libraries<jupyter_code>import warnings
warnings.filterwarnings('ignore')
%store -z<jupyter_output><empty_output><jupyter_text>Loading stored variablesIf you ran this notebook before, you may want to re-use the resources you aready created with AWS. Run the cell below to load any prevously created variables. You should see a print-out of the existing variables. If you don't see anything printed then it's probably the first time you are running the notebook!<jupyter_code>%store -r
%store<jupyter_output>Stored variables and their in-db values:<jupyter_text>**Important: You must have run the previous sequential notebooks to retrieve variables using the StoreMagic command.**  Import libraries<jupyter_code>import json
import boto3
import sagemaker<jupyter_output><empty_output><jupyter_text>Getting started: Creating Resources[overview](all-up-overview)----In order to successfully run this notebook you will need to create some AWS resources. First, an S3 bucket will be created to store all the data for this tutorial. Once created, you will then need to create an AWS Glue role using the IAM console then attach a policy to the S3 bucket to allow FeatureStore access to this notebook. If you've already run this notebook and are picking up where you left off, then running the cells below should pick up the resources you already created without creating any additional resources.  Add FeatureStore policy to Studio's execution role![title](images/iam-policies.png)1. In a separate brower tab go to the IAM section of the AWS Console2. Navigate to the Roles section and select the execution role you're using for your SageMaker Studio user    * If you're not sure what role you're using, run the cell below to print it out3. Attach the  AmazonSageMakerFeatureStoreAccess  policy to this role. Once attached, the changes take  effect immediately.  Set region, boto3 and SageMaker SDK variables<jupyter_code># You can change this to a region of your choice

region = sagemaker.Session().boto_region_name
print("Using AWS Region: {}".format(region))
boto3.setup_default_session(region_name=region)
boto_session = boto3.Session(region_name=region)
s3_client = boto3.client("s3", region_name=region)
sagemaker_client = boto_session.client("sagemaker")
sagemaker_session = sagemaker.session.Session(
    boto_session=boto_session, sagemaker_client=sagemaker_client
)
sagemaker_role = sagemaker.get_execution_role()
print(f"Sagemaker Role: {sagemaker_role}")
if 'bucket' not in locals():
    bucket = sagemaker_session.default_bucket()
    %store bucket
print(f'Bucket name: {bucket}')<jupyter_output>Stored 'bucket' (str)
Bucket name: sagemaker-us-east-1-875692608981<jupyter_text>Create a directory in the SageMaker default bucket for this tutorial<jupyter_code>if 'prefix' not in locals():
    prefix = 'fraud-detect-demo'
    %store prefix
print(f'Sagemaker Bucket: s3://{bucket}/{prefix}')<jupyter_output>Stored 'prefix' (str)
Sagemaker Bucket: s3://sagemaker-us-east-1-875692608981/fraud-detect-demo<jupyter_text>Upload raw data to S3Before you can preprocess the raw data with Data Wrangler, it must exist in S3. ![Data Prep and Store](./images/uploadrawdata.png)<jupyter_code>s3_client.upload_file(
    Filename="data/claims.csv", Bucket=bucket, Key=f"{prefix}/data/raw/claims.csv"
)
s3_client.upload_file(
    Filename="data/customers.csv", Bucket=bucket, Key=f"{prefix}/data/raw/customers.csv"
)<jupyter_output><empty_output><jupyter_text>Update attributes within the  `.flow` file DataWrangler will generate a .flow file. It contains a reference to an S3 bucket used during the Wrangling. This may be different from the one you have as a default in this notebook eg if the Wrangling was done by someone else, you will probably not have access to their bucket and you now need to point to your own S3 bucket so you can actually load the .flow file into Wrangler or access the data.After running the cell below you can open the `claims.flow` and `customers.flow` files and export the data to S3 or you can continue the guide using the provided `data/claims_preprocessed.csv` and `data/customers_preprocessed.csv` files.<jupyter_code>import string
claims_flow_template_file = "flows/claims_flow_template"

with open(claims_flow_template_file, "r") as f:
    variables = {"bucket": bucket, "prefix": prefix}
    template = string.Template(f.read())
    claims_flow = template.substitute(variables)
    claims_flow = json.loads(claims_flow)

with open("flows/claims.flow", "w") as f:
    json.dump(claims_flow, f)

customers_flow_template_file = "flows/customers_flow_template"

with open(customers_flow_template_file, "r") as f:
    variables = {"bucket": bucket, "prefix": prefix}
    template = string.Template(f.read())
    customers_flow = template.substitute(variables)
    customers_flow = json.loads(customers_flow)

with open("flows/customers.flow", "w") as f:
    json.dump(customers_flow, f)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08489795918367347, 'fraction_numerical': 0.009469387755102041, 'mean_word_length': 5.565916398713826, 'pattern_counts': {'":': 4, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '188992', 'license': 'MIT', 'n_tokens_mistral': 1861, 'n_tokens_neox': 1771, 'n_words': 682, 'path': '03a-DataPrep-Wrangler.ipynb', 'repo_name': 'yiktol/mlt', 'size': '6125'}