<jupyter_start><jupyter_text>Instructions to use in SagemakerIn order to use dask1. Activate Conda Python 3 `source activate python3`2. Install dask and s3fs `conda install dask s3fs -c conda-forge -y`<jupyter_code>import boto3
import dask.dataframe as dd
from sagemaker import get_execution_role
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from fastparquet import ParquetFile

role = get_execution_role()

pd.set_option('display.max_columns', None)
# Utility functions here

def create_s3_file_url(folder, file_name,bucket='tally-ai-dspt3'):
    return 's3://{}/{}/{}'.format(bucket, folder, file_name)<jupyter_output><empty_output><jupyter_text>Loading Yelp `business` data<jupyter_code>folder = 'yelp-kaggle-raw-data'
file_name = 'yelp_academic_dataset_business.json'
s3_business = create_s3_file_url(folder, file_name)
print(f"Fetching data from {s3_business}")
business = dd.read_json(s3_business, lines=True, blocksize=32e6)
business.head()<jupyter_output>Fetching data from s3://tally-ai-dspt3/yelp-kaggle-raw-data/yelp_academic_dataset_business.json<jupyter_text>Dropping Missing ValuesBefore dropping na in `categories`<jupyter_code>business.compute().shape<jupyter_output><empty_output><jupyter_text>After dropping na in categories<jupyter_code>business = business.dropna(subset=['categories'])
business.compute().shape<jupyter_output><empty_output><jupyter_text>List of Unique Categories<jupyter_code>business["categories"].unique().compute()<jupyter_output><empty_output><jupyter_text>Filtering `restaurants` only Before filtering for `restaurants` businesses, total unique businesses are<jupyter_code>business["categories"].unique().compute().shape
restaurants = business[business["categories"].str.contains("Restaurants")]
restaurants.head()<jupyter_output><empty_output><jupyter_text>After filtering for `restaurants` total unique businesses dropped<jupyter_code>restaurants.compute().shape<jupyter_output><empty_output><jupyter_text>Rename `stars` to `business_stars`<jupyter_code>restaurants['business_stars'] = restaurants['stars']
restaurants = restaurants.drop("stars", axis=1)
restaurants.head()<jupyter_output><empty_output><jupyter_text>Exporting  `business` as CSV to S3<jupyter_code>file_name = 'business.csv'
folder = 'yelp-restaurants'
s3_file_name = create_s3_file_url(folder, file_name)
print(f"Storing to {s3_file_name}")
restaurants.to_csv(s3_file_name, single_file = True)<jupyter_output>Storing to s3://tally-ai-dspt3/yelp-restaurants/business.csv<jupyter_text>Loading Yelp `reviews` ~ 6GB data<jupyter_code>folder = 'yelp-kaggle-raw-data'
file_name = 'yelp_academic_dataset_review.json'
s3_reviews = create_s3_file_url(folder, file_name)
print(f"Fetching data from {s3_reviews}")
reviews = dd.read_json(s3_reviews, blocksize=32e6)
reviews.head()<jupyter_output>Fetching data from s3://tally-ai-dspt3/yelp-kaggle-raw-data/yelp_academic_dataset_review.json<jupyter_text>Total Reviews before merging<jupyter_code>reviews.compute().shape<jupyter_output><empty_output><jupyter_text>Rename `stars` to `reviews_stars`<jupyter_code>reviews['reviews_stars'] = reviews['stars']
reviews = reviews.drop("stars", axis=1)
reviews.head()<jupyter_output><empty_output><jupyter_text>Merging `business` and `reviews` data<jupyter_code>restaurants_reviews = restaurants.merge(reviews, on="business_id",  how = 'inner')
restaurants_reviews.head()<jupyter_output><empty_output><jupyter_text>Total Restaurant Reviews after merging<jupyter_code>restaurants_reviews.compute().shape<jupyter_output><empty_output><jupyter_text>Filtering out columns from Restaurant Reviews<jupyter_code>cols_to_keep = ["review_id", "business_id", "reviews_stars", "date", "text"]
restaurants_reviews = restaurants_reviews[cols_to_keep]
restaurants_reviews.head()<jupyter_output><empty_output><jupyter_text>NOTE: Saving the `restaurants_reviews` dask dataframe to `.csv` file failed due to large size.  Exporting `reviews` as fastparquet to S3<jupyter_code>file_name = 'reviews.parquet.gzip'
folder = 'yelp-restaurants'
s3_file_name = create_s3_file_url(folder, file_name)
print(f"Storing to {s3_file_name}")
restaurants_reviews.to_parquet(s3_file_name,  compression="gzip")<jupyter_output>Storing to s3://tally-ai-dspt3/yelp-restaurants/reviews.parquet.gzip
{'fraction_non_alphanumeric': 0.1176608187134503, 'fraction_numerical': 0.010526315789473684, 'mean_word_length': 10.282321899736148, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 61, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '186814', 'license': 'MIT', 'n_tokens_mistral': 1599, 'n_tokens_neox': 1557, 'n_words': 240, 'path': 'notebooks/Harsh-yelp-review-etl.ipynb', 'repo_name': 'BloomTech-Labs/tally-ai-ds', 'size': '4275'}