<jupyter_start><jupyter_text>Extract and transform Yelp dataset  The below command gets a utility script which is used to download files from Google Drive<jupyter_code>!sudo wget -O /usr/sbin/gdrivedl 'https://f.mjh.nz/gdrivedl'
!sudo chmod +x /usr/sbin/gdrivedl<jupyter_output>--2019-07-01 06:31:43--  https://f.mjh.nz/gdrivedl
Resolving f.mjh.nz (f.mjh.nz)... 192.168.3.11, 192.168.3.11, fd00:c2b6:b24b:be67:2827:688d:e6a1:6a3b:1ee9, ...
Connecting to f.mjh.nz (f.mjh.nz)|192.168.3.11|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1362 (1.3K) [application/octet-stream]
Saving to: ‘/usr/sbin/gdrivedl’

/usr/sbin/gdrivedl  100%[===================>]   1.33K  --.-KB/s    in 0s      

2019-07-01 06:31:43 (26.6 MB/s) - ‘/usr/sbin/gdrivedl’ saved [1362/1362]<jupyter_text>Download Yelp dataset from Google Drive<jupyter_code>!gdrivedl https://drive.google.com/file/d/12d2zEuuw8-k5fRclu0vcjR2N6sTmReVz/view?usp=sharing /home/jovyan/yelp_dataset.tar<jupyter_output>File ID: 12d2zEuuw8-k5fRclu0vcjR2N6sTmReVz
Downloading: https://docs.google.com/uc?export=download&id=12d2zEuuw8-k5fRclu0vcjR2N6sTmReVz > /home/jovyan/yelp_dataset.tar.47.file
Downloading: https://docs.google.com/uc?export=download&id=12d2zEuuw8-k5fRclu0vcjR2N6sTmReVz&confirm=J67i > /home/jovyan/yelp_dataset.tar.47.file
Moving: /home/jovyan/yelp_dataset.tar.47.file > /home/jovyan/yelp_dataset.tar
Saved: /home/jovyan/yelp_dataset.tar
DONE!<jupyter_text>Extract JSON files<jupyter_code>!mkdir /home/jovyan/logs
!mkdir /home/jovyan/yelp_dataset
!tar -C /home/jovyan/yelp_dataset -xvf /home/jovyan/yelp_dataset.tar<jupyter_output>business.json
checkin.json
review.json
tip.json
user.json
photo.json
Dataset_Challenge_Dataset_Agreement.pdf
Yelp_Dataset_Challenge_Round_13.pdf<jupyter_text>Remove the tar file after extracting to save space<jupyter_code>!rm /home/jovyan/yelp_dataset.tar<jupyter_output><empty_output><jupyter_text>Below are the set of Spark jobs run locally to transform JSON files to Parquet format. Can be submitted to YARN or other cluster by proving `--master ` and `--deploy-mode ` to `spark-submit`.<jupyter_code>!spark-submit utils/convert_json_to_parquet.py /home/jovyan/yelp_dataset/business.json \
                                               /home/jovyan/yelp_dataset_parquet/business \
                                               --numeric_id_over_col business_id > /home/jovyan/logs/business.txt 2>&1
!spark-submit --driver-memory 8G utils/convert_json_to_parquet.py /home/jovyan/yelp_dataset/user.json \
                                                                  /home/jovyan/yelp_dataset_parquet/user \
                                                                  --numeric_id_over_col user_id > /home/jovyan/logs/user.txt 2>&1
!spark-submit --driver-memory 8G utils/convert_json_to_parquet.py /home/jovyan/yelp_dataset/review.json \
                                                                  /home/jovyan/yelp_dataset_parquet/review \
                                                                  --numeric_id_over_col review_id > /home/jovyan/logs/review.txt 2>&1
!spark-submit --driver-memory 3G utils/convert_json_to_parquet.py /home/jovyan/yelp_dataset/checkin.json \
                                                                  /home/jovyan/yelp_dataset_parquet/checkin > /home/jovyan/logs/checkin.txt 2>&1
!spark-submit --driver-memory 3G utils/convert_json_to_parquet.py /home/jovyan/yelp_dataset/tip.json \
                                                                  /home/jovyan/yelp_dataset_parquet/tip  > /home/jovyan/logs/tip.txt 2>&1
!spark-submit utils/convert_json_to_parquet.py /home/jovyan/yelp_dataset/photo.json \
                                               /home/jovyan/yelp_dataset_parquet/photo \
                                               --numeric_id_over_col photo_id > /home/jovyan/logs/photo.txt 2>&1<jupyter_output><empty_output><jupyter_text>Remove the JSON files to save space<jupyter_code>!rm -r /home/jovyan/yelp_dataset<jupyter_output><empty_output><jupyter_text>Download Yelp photos dataset if necessary. (not needed right now)<jupyter_code>!gdrivedl https://drive.google.com/file/d/1npepf7QsFVFZLWrewUaLR6Hh6fBV_yRF/view?usp=sharing /home/jovyan/yelp_photos.tar<jupyter_output>File ID: 1npepf7QsFVFZLWrewUaLR6Hh6fBV_yRF
Downloading: https://docs.google.com/uc?export=download&id=1npepf7QsFVFZLWrewUaLR6Hh6fBV_yRF > /home/jovyan/yelp_photos.tar.4299.file
Downloading: https://docs.google.com/uc?export=download&id=1npepf7QsFVFZLWrewUaLR6Hh6fBV_yRF&confirm=JAof > /home/jovyan/yelp_photos.tar.4299.file
Moving: /home/jovyan/yelp_photos.tar.4299.file > /home/jovyan/yelp_photos.tar
Saved: /home/jovyan/yelp_photos.tar
DONE!<jupyter_text>Extract files from tar file<jupyter_code>!mkdir /home/jovyan/yelp_photos
!tar -C /home/jovyan/yelp_photos -xf /home/jovyan/yelp_photos.tar<jupyter_output><empty_output><jupyter_text>Remove the tar file to save space<jupyter_code>!rm /home/jovyan/yelp_photos.tar<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11694510739856802, 'fraction_numerical': 0.03818615751789976, 'mean_word_length': 3.0556451612903226, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 52, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 3, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 31, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '375017', 'license': 'MIT', 'n_tokens_mistral': 2068, 'n_tokens_neox': 1894, 'n_words': 247, 'path': 'Extracting Yelp Dataset.ipynb', 'repo_name': 'NandeeshHD/yelp-data-challenge', 'size': '5010'}