<jupyter_start><jupyter_text>DatasetsIn this notebook, we'll take a look at preparing datasets for machinelearning using AWS Glue and create schemas to enforce validity of thedata in later stages.   When creating the AWS CloudFormation stack, a collection of syntheticdatasets were generated and stored in our solution Amazon S3 bucket witha prefix of `dataset`. Most of the features contained in these datasetsare based on the [German CreditDataset](http://archive.ics.uci.edu/ml/datasets/statlog%2B%28german%2Bcredit%2Bdata%29)(UCI Machine Learning Repository), but there are some synthetic datafields too. All personal information was generated using[`Faker`](https://faker.readthedocs.io/en/master/). We have 3 datasets intotal: credits, people and contacts.  Dataset 1: CreditsOur credits dataset contains features directly related to the creditapplication.It is a CSV file (i.e. Comma Seperated Value file) that has a header rowwith feature names. Of particular note is the feature called `default`.It is our target variable that we're trying to predict with our LightGBMmodel. We show the first two rows of the dataset below:```"credit_id","person_id","amount","duration","purpose","installment_rate","guarantor","coapplicant","default""51829372","f032303d",1169,6,"electronics",4,0,0,False```  Dataset 2: PeopleOur credits data contains features related to the people making thecredit applications (i.e. the applicants).It's a [JSON Lines](http://jsonlines.org/) file, where each row is aseparate JSON object. Of particular note is the feature called`person_id`. You'll notice that this feature was also included in thecredits dataset. It is used to connect the credit application with theapplicant. We show the first row of the dataset below:```{    "person_id": "f032303d",    "finance": {        "accounts": {            "checking": {                "balance": "negative"            }        },        "repayment_history": "very_poor",        "credits": {            "this_bank": 2,            "other_banks": 0,            "other_stores": 0        },        "other_assets": "real_estate"    },    "personal": {        "age": 67,        "gender": "male",        "relationship_status": "single",        "name": "<NAME>"    },    "dependents": [        {            "gender": "male",            "name": "<NAME>"        }    ],    "employment": {        "type": "professional",        "title": "Learning disability nurse",        "duration": 11,        "permit": "foreign"    },    "residence": {        "type": "own",        "duration": 4    }}```  Dataset 3: ContactsOur contacts dataset contains contact information for the applicants.It is a CSV file that has a header row with feature names. Once again wehave `person_id`. We show the first two rows of the dataset below:```"contact_id","person_id","type","value""5996e20a","f032303d","telephone","(716)406-9514x345"```  AWS GlueOne of the most time consuming tasks in developing a machine learningworkflow is data preperation. AWS Glue can be used to simplify thisprocess. As a demonstration of how it can be used to infer data schemasand perform extract, transform and load (ETL) jobs in Spark, we'llprepare a dataset using AWS Glue. Although our sample datasets are small,there are many real world scenarios that will benefit from thescalability of AWS Glue.When creating the AWS CloudFormation stack, a number of AWS Glue resourceswere created:* A  [Database](https://docs.aws.amazon.com/glue/latest/dg/define-database.html)  is used to organize solution's tables.* A [Crawler](https://docs.aws.amazon.com/glue/latest/dg/add-crawler.html) is  used infer formats and schemas of the datasets above.* A [Custom  Classifier](https://docs.aws.amazon.com/glue/latest/dg/add-classifier.html)  is used to help the classifier infer the schema of the contacts datasets.    * All fields are of type 'string', so we need to indicate that the first      row is a header row rather than data.* A [Job](https://docs.aws.amazon.com/glue/latest/dg/author-job.html) is used  to join the datasets together, drop certain feature, create other features,  and split train and test sets.* A  [Workflow](https://docs.aws.amazon.com/glue/latest/dg/orchestrate-using-workflows.html)  (and associated  [triggers](https://docs.aws.amazon.com/glue/latest/dg/trigger-job.html)) to  orchestrate the above crawler and job.You can explore the service console for AWS Glue for more details. We start by importing a variety of packages that will be used throughoutthe notebook. One of the most important packages used throughout thissolution is the Amazon SageMaker Python SDK (i.e. `import sagemaker`). Wealso import modules from our own custom package that can be found at`./package`.<jupyter_code>import boto3
from pathlib import Path
import sagemaker

from package import config, utils
from package.data import glue<jupyter_output><empty_output><jupyter_text>We'll now start the AWS Glue workflow which will first crawl the datasetsstored in Amazon S3 and then execute a job for our data ELT (i.e.Extract, Transform & Load).<jupyter_code>glue_run_id = glue.start_workflow(config.GLUE_WORKFLOW)<jupyter_output><empty_output><jupyter_text>Our workflow takes around 10 minutes to complete. Most of this time is spendon resource provisioning, but there is a [previewfeature](https://pages.awscloud.com/glue-reduced-spark-times-preview-2020.html)for reduced start times. We'll wait until the AWS Glue workflow has completedbefore continuing. We need the dataset before training our model in AmazonSageMaker.<jupyter_code>glue.wait_for_workflow_finished(config.GLUE_WORKFLOW, glue_run_id)<jupyter_output><empty_output><jupyter_text>With our AWS Glue workflow complete, we should now have 4 additionaldatasets in our solution's Amazon S3 bucket: `data_train`, `label_train`,`data_test` and `label_test`. We show an example first record of`data_train` below:```{	"credit__coapplicant": 0,	"credit__guarantor": 0,	"finance__credits__other_stores": 0,	"employment__permit": "foreign",	"employment__type": "labourer",	"finance__accounts__checking__balance": "no_account",	"finance__credits__this_bank": 2,	"finance__credits__other_banks": 1,	"personal__num_dependents": 1,	"finance__accounts__savings__balance": "very_high",	"residence__duration": 2,	"credit__amount": 250,	"contact__has_telephone": false,	"employment__duration": 4,	"credit__duration": 6,	"finance__repayment_history": "very_poor",	"finance__other_assets": "real_estate",	"credit__purpose": "new_car",	"residence__type": "own",	"credit__installment_rate": 2}```We now have 20 features that describe a credit application and itsapplicant. Since we are using JSON formatted record we still have thefeature names, but additional information such as the feature types canbe retrieved from the schema stored in our AWS Glue catalog. Since we'reinterested in explaining the model predictions, and our explanations willassign contributions to features, it's useful if our feature names areunderstandable.**Advanced**: We can also organize features in a hierarchy (using a seperatorin the feature names), which enables summarization of the explanations. As anexample, `employment__type` and `employment__duration` are both `employment`related features. We use two consecutive underscores (`__`) as our levelseparator.  SchemaSchemas can be used to keep track of feature names, descriptions andtypes. Our solution uses[`jsonschema`](https://python-jsonschema.readthedocs.io/en/stable/) asthe primary schema format. We have the added bonus of being able to useschemas to validate input to the trained model and deployed endpoints,and consistently map unordered dictionaries to ordered lists (as requiredby the model). We don't need to create schemas to use Amazon SageMaker,but they help in this solution.We already have most of this schema information in our AWS Glue catalog(it was added when data was exported by the AWS Glue Job), so let's startby retrieving the table schema for `data_train`.<jupyter_code>data_schema = glue.get_table_schema(
    database_name=config.GLUE_DATABASE, table_name="data_train"
)<jupyter_output><empty_output><jupyter_text>We can now add additional information such as feature descriptions, that willbe shown inside the tooltip on the visuals later on.<jupyter_code># flake8: noqa: E501
data_schema.title = "Credit Application"
data_schema.description = "An array of items used to describe a credit application."
item_descriptions_dict = {
    "contact__has_telephone": "Customer has a registered telephone number.",
    "credit__amount": "Amount of money requested as part of credit application (in EUR).",
    "credit__coapplicant": "Co-applicant on credit application.",
    "credit__duration": "Amount of time the credit is requested for (in months).",
    "credit__guarantor": "Guarantor on credit application.",
    "credit__installment_rate": "Credit installment rate (as a percentage of the customer's disposable income).",
    "credit__purpose": "Customer's reason for requiring credit.",
    "employment__duration": "Amount of time the customer has been employed at their current employer (in years).",
    "employment__permit": "Customer's current work permit type.",
    "employment__type": "Customer's current job classification.",
    "finance__accounts__checking__balance": "Customer's checking account balance.",
    "finance__accounts__savings__balance": "Customer's savings account balance.",
    "finance__credits__other_banks": "Count of credits the customer has at other banks.",
    "finance__credits__other_stores": "Count of credits the customer has at other stores.",
    "finance__credits__this_bank": "Count of credits the customer has at this bank.",
    "finance__other_assets": "Customer's most significant asset.",
    "finance__repayment_history": "Quality of the customer's repayment history.",
    "personal__num_dependents": "Count of the customer's dependents.",
    "residence__duration": "Amount of time the customer has been at their current residence (in years).",
    "residence__type": "Class of the customer's residence."
}
data_schema.item_descriptions_dict = item_descriptions_dict<jupyter_output><empty_output><jupyter_text>We do the same for `label_train` too.<jupyter_code>label_schema = glue.get_table_schema(
    database_name=config.GLUE_DATABASE, table_name="label_train"
)
label_schema.title = "Credit Application Outcome"
item_descriptions_dict = {
    "credit__default": (
        "0 if the customer successfully made credit payments, "
        "1 if the customer defaulted on credit payments.")
}
label_schema.item_descriptions_dict = item_descriptions_dict<jupyter_output><empty_output><jupyter_text>Since the schemas for train and test datasets are the same, we can skip`data_test` and `label_test`.We can save our updated schemas to disk, in preperation for uploading toAmazon S3. You can check the `schema_folder` afterwards, and examine the`data.schema.json` and `label.schema.json` files for data and labelschemas respectively.<jupyter_code>current_folder = utils.get_current_folder(globals())
schema_folder = Path(current_folder, "../schemas").resolve()
data_schema_filepath = Path(schema_folder, "data.schema.json")
data_schema.save(data_schema_filepath)
label_schema_filepath = Path(schema_folder, "label.schema.json")
label_schema.save(label_schema_filepath)<jupyter_output><empty_output><jupyter_text>Up next, we create a SageMaker Session. A SageMaker Session can be used toconveniently perform certain AWS actions, such as uploading and downloadingfiles from Amazon S3. We use the SageMaker Session to upload our schemas toAmazon S3.<jupyter_code>boto_session = boto3.session.Session(region_name=config.AWS_REGION)
sagemaker_session = sagemaker.Session(boto_session)

sagemaker_session.upload_data(
    path=str(schema_folder),
    bucket=config.S3_BUCKET,
    key_prefix=config.SCHEMAS_S3_PREFIX
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08964996222613951, 'fraction_numerical': 0.009233610341643583, 'mean_word_length': 5.29039070749736, 'pattern_counts': {'":': 68, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 9, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '188108', 'license': 'Apache-2.0', 'n_tokens_mistral': 3475, 'n_tokens_neox': 3259, 'n_words': 1220, 'path': 'sagemaker/notebooks/1_datasets.ipynb', 'repo_name': 'kziemski/sagemaker-explaining-credit-decisions', 'size': '11927'}