<jupyter_start><jupyter_text>Prepare Dataset for Model Training and Evaluating  Amazon Customer Reviews Datasethttps://s3.amazonaws.com/amazon-reviews-pds/readme.html Schema- `marketplace`: 2-letter country code (in this case all "US").- `customer_id`: Random identifier that can be used to aggregate reviews written by a single author.- `review_id`: A unique ID for the review.- `product_id`: The Amazon Standard Identification Number (ASIN).  `http://www.amazon.com/dp/` links to the product's detail page.- `product_parent`: The parent of that ASIN.  Multiple ASINs (color or format variations of the same product) can roll up into a single parent.- `product_title`: Title description of the product.- `product_category`: Broad product category that can be used to group reviews (in this case digital videos).- `star_rating`: The review's rating (1 to 5 stars).- `helpful_votes`: Number of helpful votes for the review.- `total_votes`: Number of total votes the review received.- `vine`: Was the review written as part of the [Vine](https://www.amazon.com/gp/vine/help) program?- `verified_purchase`: Was the review from a verified purchase?- `review_headline`: The title of the review itself.- `review_body`: The text of the review.- `review_date`: The date the review was written.<jupyter_code>import boto3
import sagemaker
import pandas as pd

sess   = sagemaker.Session()
bucket = sess.default_bucket()
role = sagemaker.get_execution_role()
region = boto3.Session().region_name<jupyter_output><empty_output><jupyter_text>DownloadLet's start by retrieving a subset of the Amazon Customer Reviews dataset.<jupyter_code>!aws s3 cp 's3://amazon-reviews-pds/tsv/amazon_reviews_us_Digital_Software_v1_00.tsv.gz' ./data/
import csv

df = pd.read_csv('./data/amazon_reviews_us_Digital_Software_v1_00.tsv.gz', 
                 delimiter='\t', 
                 quoting=csv.QUOTE_NONE,
                 compression='gzip')
df.shape
df.head(5)
import matplotlib.pyplot as plt
%matplotlib inline
%config InlineBackend.figure_format='retina'

df[['star_rating', 'review_id']].groupby('star_rating').count().plot(kind='bar', title='Breakdown by Star Rating')
plt.xlabel('Star Rating')
plt.ylabel('Review Count')<jupyter_output><empty_output><jupyter_text>Balance the Dataset<jupyter_code>from sklearn.utils import resample

five_star_df = df.query('star_rating == 5')
four_star_df = df.query('star_rating == 4')
three_star_df = df.query('star_rating == 3')
two_star_df = df.query('star_rating == 2')
one_star_df = df.query('star_rating == 1')

# Check which sentiment has the least number of samples
minority_count = min(five_star_df.shape[0], 
                     four_star_df.shape[0], 
                     three_star_df.shape[0], 
                     two_star_df.shape[0], 
                     one_star_df.shape[0]) 

five_star_df = resample(five_star_df,
                        replace = False,
                        n_samples = minority_count,
                        random_state = 27)

four_star_df = resample(four_star_df,
                        replace = False,
                        n_samples = minority_count,
                        random_state = 27)

three_star_df = resample(three_star_df,
                        replace = False,
                        n_samples = minority_count,
                        random_state = 27)

two_star_df = resample(two_star_df,
                        replace = False,
                        n_samples = minority_count,
                        random_state = 27)

one_star_df = resample(one_star_df,
                        replace = False,
                        n_samples = minority_count,
                        random_state = 27)

df_balanced = pd.concat([five_star_df, four_star_df, three_star_df, two_star_df, one_star_df])
df_balanced = df_balanced.reset_index(drop=True)

df_balanced.shape
df_balanced[['star_rating', 'review_id']].groupby('star_rating').count().plot(kind='bar', title='Breakdown by Star Rating')
plt.xlabel('Star Rating')
plt.ylabel('Review Count')
df_balanced.head(5)<jupyter_output><empty_output><jupyter_text>Split the Data into Train, Validation, and Test Sets<jupyter_code>from sklearn.model_selection import train_test_split

# Split all data into 90% train and 10% holdout
df_train, df_holdout = train_test_split(df_balanced, 
                                        test_size=0.10,
                                        stratify=df_balanced['star_rating'])

# Split holdout data into 50% validation and 50% test
df_validation, df_test = train_test_split(df_holdout,
                                          test_size=0.50, 
                                          stratify=df_holdout['star_rating'])

# Pie chart, where the slices will be ordered and plotted counter-clockwise:
labels = ['Train', 'Validation', 'Test']
sizes = [len(df_train.index), len(df_validation.index), len(df_test.index)]
explode = (0.1, 0, 0)  

fig1, ax1 = plt.subplots()

ax1.pie(sizes, explode=explode, labels=labels, autopct='%1.1f%%', startangle=90)

# Equal aspect ratio ensures that pie is drawn as a circle.
ax1.axis('equal')  

plt.show()<jupyter_output><empty_output><jupyter_text>Show 90% Train Data Split<jupyter_code>df_train.shape
df_train[['star_rating', 'review_id']].groupby('star_rating').count().plot(kind='bar', title='90% Train Breakdown by Star Rating')<jupyter_output><empty_output><jupyter_text>Show 5% Validation Data Split<jupyter_code>df_validation.shape
df_validation[['star_rating', 'review_id']].groupby('star_rating').count().plot(kind='bar', title='5% Validation Breakdown by Star Rating')<jupyter_output><empty_output><jupyter_text>Show 5% Test Data Split<jupyter_code>df_test.shape
df_test[['star_rating', 'review_id']].groupby('star_rating').count().plot(kind='bar', title='5% Test Breakdown by Star Rating')<jupyter_output><empty_output><jupyter_text>Select `star_rating` and `review_body` for Training<jupyter_code>df_train = df_train[['star_rating', 'review_body']]
df_train.shape
df_train.head(5)<jupyter_output><empty_output><jupyter_text>Convert `star_rating` into label ids (0-4)<jupyter_code>def to_id(rating):
    rating = int(rating)
    if rating == 1:
        return 0
    elif rating == 2:
        return 1
    elif rating == 3:
        return 2
    elif rating == 4:
        return 3
    elif rating == 5:
        return 4
df_train['star_rating_id'] = df_train.star_rating.apply(to_id)
df_train = df_train[['star_rating_id', 'review_body']]
df_train.shape
df_train.head()<jupyter_output><empty_output><jupyter_text>Write a CSV With No Header for Comprehend<jupyter_code>jumpstart_train_path = './data.csv'
df_train.to_csv(jumpstart_train_path, index=False, header=False)<jupyter_output><empty_output><jupyter_text>Upload Train Data to S3 for Comprehend<jupyter_code>train_s3_prefix = 'data'
jumpstart_train_s3_uri = sess.upload_data(path=jumpstart_train_path, key_prefix=train_s3_prefix)
jumpstart_train_s3_uri
!aws s3 ls $jumpstart_train_s3_uri<jupyter_output>2020-12-09 15:33:52   13657516 amazon_reviews_us_Digital_Software_v1_00_comprehend.csv<jupyter_text>Store the location of our train data in our notebook server to be used next<jupyter_code>%store jumpstart_train_s3_uri
%store
%%javascript
Jupyter.notebook.save_checkpoint();
Jupyter.notebook.session.delete();<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.108348134991119, 'fraction_numerical': 0.015849159721273397, 'mean_word_length': 3.8508946322067596, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 48, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '311122', 'license': 'Apache-2.0', 'n_tokens_mistral': 2466, 'n_tokens_neox': 2303, 'n_words': 533, 'path': '07_train/jumpstart/Finetune_BERT_Jumpstart.ipynb', 'repo_name': 'MarcusFra/workshop', 'size': '7319'}