<jupyter_start><jupyter_text>Copy TSV Data To S3  We have chosen the [Amazon Customer Reviews Dataset](https://s3.amazonaws.com/amazon-reviews-pds/readme.html) as our main dataset.The dataset is shared in a public Amazon S3 bucket, and is available in two file formats: * Tab separated value (TSV), a text format - `s3://amazon-reviews-pds/tsv/`* Parquet, an optimized columnar binary format - `s3://amazon-reviews-pds/parquet/`The Parquet dataset is partitioned (divided into subfolders) by the column `product_category` to further improve query performance. With this, you can use a `WHERE` clause on product_category in your SQL queries to only read data specific to that category.We can use the AWS Command Line Interface (CLI) to list the S3 bucket content using the following CLI commands:<jupyter_code>!aws s3 ls s3://amazon-reviews-pds/tsv/
!aws s3 ls s3://amazon-reviews-pds/parquet/<jupyter_output>PRE product_category=Apparel/
                           PRE product_category=Automotive/
                           PRE product_category=Baby/
                           PRE product_category=Beauty/
                           PRE product_category=Books/
                           PRE product_category=Camera/
                           PRE product_category=Digital_Ebook_Purchase/
                           PRE product_category=Digital_Music_Purchase/
                           PRE product_category=Digital_Software/
                           PRE product_category=Digital_Video_Download/
                           PRE product_category=Digital_Video_Games/
                           PRE product_category=Electronics/
                           PRE product_category=Furniture/
                           PRE product_category=Gift_Card/
                           PRE product_category=Grocery/
                           PRE product_category=Health_&_Personal_Care/
   [...]<jupyter_text>To Simulate an Application Writing Into Our Data Lake, We Copy the Public TSV Dataset to a Private S3 Bucket in our Account<jupyter_code>import boto3
import sagemaker
import pandas as pd

sess = sagemaker.Session()
bucket = sess.default_bucket()
role = sagemaker.get_execution_role()
region = boto3.Session().region_name
account_id = boto3.client("sts").get_caller_identity().get("Account")

sm = boto3.Session().client(service_name="sagemaker", region_name=region)<jupyter_output><empty_output><jupyter_text>Set S3 Source Location (Public S3 Bucket)<jupyter_code>s3_public_path_tsv = "s3://amazon-reviews-pds/tsv"
%store s3_public_path_tsv<jupyter_output>Stored 's3_public_path_tsv' (str)<jupyter_text>Set S3 Destination Location (Our Private S3 Bucket)<jupyter_code>s3_private_path_tsv = "s3://{}/amazon-reviews-pds/tsv".format(bucket)
print(s3_private_path_tsv)
%store s3_private_path_tsv<jupyter_output>Stored 's3_private_path_tsv' (str)<jupyter_text>Copy Data From the Public S3 Bucket to our Private S3 Bucket in this AccountAs the full dataset is pretty large, let's just copy 3 files into our bucket to speed things up later.<jupyter_code>!aws s3 cp --recursive $s3_public_path_tsv/ $s3_private_path_tsv/ --exclude "*" --include "amazon_reviews_us_Digital_Software_v1_00.tsv.gz"
!aws s3 cp --recursive $s3_public_path_tsv/ $s3_private_path_tsv/ --exclude "*" --include "amazon_reviews_us_Digital_Video_Games_v1_00.tsv.gz"
!aws s3 cp --recursive $s3_public_path_tsv/ $s3_private_path_tsv/ --exclude "*" --include "amazon_reviews_us_Gift_Card_v1_00.tsv.gz"<jupyter_output>copy: s3://amazon-reviews-pds/tsv/amazon_reviews_us_Digital_Software_v1_00.tsv.gz to s3://sagemaker-us-east-1-117859797117/amazon-reviews-pds/tsv/amazon_reviews_us_Digital_Software_v1_00.tsv.gz
copy: s3://amazon-reviews-pds/tsv/amazon_reviews_us_Digital_Video_Games_v1_00.tsv.gz to s3://sagemaker-us-east-1-117859797117/amazon-reviews-pds/tsv/amazon_reviews_us_Digital_Video_Games_v1_00.tsv.gz
copy: s3://amazon-reviews-pds/tsv/amazon_reviews_us_Gift_Card_v1_00.tsv.gz to s3://sagemaker-us-east-1-117859797117/amazon-reviews-pds/tsv/amazon_reviews_us_Gift_Card_v1_00.tsv.gz<jupyter_text>_Make sure ^^^^ this ^^^^ S3 COPY command above runs succesfully. We will need those datafiles for the rest of this workshop._  List Files in our Private S3 Bucket in this Account<jupyter_code>print(s3_private_path_tsv)
!aws s3 ls $s3_private_path_tsv/
from IPython.core.display import display, HTML

display(
    HTML(
        '<b>Review <a target="blank" href="https://s3.console.aws.amazon.com/s3/buckets/sagemaker-{}-{}/amazon-reviews-pds/?region={}&tab=overview">S3 Bucket</a></b>'.format(
            region, account_id, region
        )
    )
)<jupyter_output><empty_output><jupyter_text>Store Variables for the Next Notebooks<jupyter_code>%store<jupyter_output><empty_output><jupyter_text>Release Resources<jupyter_code>%%html

<p><b>Shutting down your kernel for this notebook to release resources.</b></p>
<button class="sm-command-button" data-commandlinker-command="kernelmenu:shutdown" style="display:none;">Shutdown Kernel</button>
        
<script>
try {
    els = document.getElementsByClassName("sm-command-button");
    els[0].click();
}
catch(err) {
    // NoOp
}    
</script>
%%javascript

try {
    Jupyter.notebook.save_checkpoint();
    Jupyter.notebook.session.delete();
}
catch(err) {
    // NoOp
}
# Internal - DO NOT RUN

# step_prefix = '04_prepare'
# !aws s3 cp --recursive $s3_public_path_tsv/ s3://dsoaws/$step_prefix/ --exclude "*" --include "amazon_reviews_us_Digital_Software_v1_00.tsv.gz"
# !aws s3 cp --recursive $s3_public_path_tsv/ s3://dsoaws/$step_prefix/ --exclude "*" --include "amazon_reviews_us_Digital_Video_Games_v1_00.tsv.gz"
# !aws s3 cp --recursive $s3_public_path_tsv/ s3://dsoaws/$step_prefix/ --exclude "*" --include "amazon_reviews_us_Gift_Card_v1_00.tsv.gz"
# !aws s3 ls --recursive s3://dsoaws/$step_prefix/<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10583817002389895, 'fraction_numerical': 0.024411061795834757, 'mean_word_length': 3.902928870292887, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '82988', 'license': 'Apache-2.0', 'n_tokens_mistral': 2128, 'n_tokens_neox': 2010, 'n_words': 402, 'path': '01-01_Copy_TSV_To_S3.ipynb', 'repo_name': 'tuhinsharma121/aws-sagemaker-quickstart', 'size': '5858'}