<jupyter_start><jupyter_text>**Steps to download the data*** We download the set of Amazon reviews from the website (https://nijianmo.github.io/amazon/index.html). We use the *Small subsets for experimentation* and not the Complete review data. They are referred to as k-core and are available for several categories. We only download the reviews (5-core) and not the ratings.* The list of categories is specified in the file - *productCategories.txt*. Even though this is a *small* subset, the reviews for some of the categories like Books can be quite large (6.6GB). We selected the following categories to balance the size and diversity of the categories.        * AMAZON_FASHION        * All_Beauty        * Appliances        * Arts_Crafts_and_Sewing        * Automotive        * Cell_Phones_and_Accessories        * Digital_Music        * Gift_Cards        * Grocery_and_Gourmet_Food        * Industrial_and_Scientific        * Luxury_Beauty        * Magazine_Subscriptions        * Musical_Instruments        * Office_Products        * Patio_Lawn_and_Garden        * Prime_Pantry        * Software* If you would like to change them, you need to add/delete entries to the *productCategories.txt** Once the set of categories is set, you need to run the provided shell script *getData.sh* by executing the command `sh getData.sh`. This downloads each category file into the current folder* Once this script completes running, we can run the cells that follow in this notebook to create the final dataset used in the chapter<jupyter_code>import pandas as pd
from nltk.tokenize import sent_tokenize, word_tokenize

def num_sentences(text):
    text = str(text)
    return len(sent_tokenize(text))

def get_category_data(cat_name):
    # Load the zip file and deduplicate
    data = pd.read_json(cat_name+'_5.json.gz', lines=True)
    data = data.drop(columns=['style','reviewerName','vote','image'])
    print ('Original Number of Reviews in ', cat_name, ' - ', len(data))
    data = data.drop_duplicates()
    print ('Number of reviews after de-duplication - ', len(data))
        
    # Keep reviews that contain only one sentence (remove if no text found)
    data['num_sentences'] = data['reviewText'].apply(num_sentences)
    data = data[data['num_sentences'] == 1]
    data = data[~data['reviewText'].isna()]
    data = data.drop(columns=['num_sentences'])
    print ('Number of reviews with only one sentence - ', len(data))
    
    return data
# Create list of product categories
with open('productCategories.txt', 'r') as f:
    productCategories = []
    for line in f:
        if line.strip() != '':
            productCategories.append(line.rstrip('\n'))
# Load reviews for each category and append all to create dataset
finalDF = pd.DataFrame(columns=['overall', 'verified', 'reviewTime', 'reviewerID', 'asin', 'reviewText', 'summary', 'unixReviewTime'])
for category in productCategories:
    returnDF = get_category_data(category)
    finalDF = finalDF.append(returnDF)
len(finalDF)
finalDF['overall'].value_counts()
# We intend to use rating scores to tag positive and negative sentiment
# As a result a score of 3 is not very helpful and we drop all reviews with this rating from the data
finalDF = finalDF[finalDF['overall'] != 3]
# We tag all the reviews with a high rating [4,5] as positive sentiment (1)
# We tag all the reviews with a low rating [1,2] as negative sentiment (0)

finalDF['sentiment'] = 0
finalDF.loc[finalDF['overall'] > 3, 'sentiment'] = 1
finalDF.loc[finalDF['overall'] < 3, 'sentiment'] = 0
# To maintain a reasonable size of the dataset that we can work with easily while still capturing enough
# observations from both samples, we sample 150k observations from each group
def sampling_k_elements(group, k=150000):
    if len(group) < k:
        return group
    return group.sample(k, random_state=42)

finalDF = finalDF.groupby('sentiment').apply(sampling_k_elements).reset_index(drop=True)
print (finalDF['sentiment'].value_counts())
finalDF = finalDF.drop(columns=['sentiment'])
finalDF.to_json('reviews_5_balanced.json', orient='records', lines=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09590699927343183, 'fraction_numerical': 0.007265681763138775, 'mean_word_length': 4.181932245922209, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 6, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '605668', 'license': 'Apache-2.0', 'n_tokens_mistral': 1198, 'n_tokens_neox': 1161, 'n_words': 431, 'path': 'data/amazon-product-reviews/Data Preparation.ipynb', 'repo_name': 'oooooolr/blueprints-text', 'size': '4129'}