<jupyter_start><jupyter_text>Data extraction notebook**Author: <NAME>** May 2019 | Chicago, IL.  Table of contents- [Overview](ov)- [Importing libraries](imp)- [Convert from Json to CSV](conv)  OverviewThis notebook contains the code to convert the JSON files of the Yelp Academic Dataset into a CSV format. This is performed in order to read and manipulate the data in pandas and numpy for modelling purposes. The output of this notebook will be used in subsequent notebooks of this repository.   Importing librariesIn this section we outline the initial code needed to run this workbook. If this code returns an error we recommend the reader to verify that the most up to date version of the libraries mentioned below have been installed in their computers. For a guideline on python installation of modules please refer to the __[official documentation](https://docs.python.org/3/installing/)__.<jupyter_code>#Importing relevant libraries
import pandas as pd
import numpy as np
import json<jupyter_output><empty_output><jupyter_text>Convert from JSON to CSV The following lines of code convert the JSON files from the Yelp Academic Dataset into CSV files that will be used for modelling in subsequent notebooks. Business dataset<jupyter_code>#Open and read the json file
with open('./yelp_dataset/business.json', 'rb') as f:
    data = f.readlines()

#Convert each row of the Json file into a dictionary
data = [eval(x.decode().rstrip().replace('null','None')) for x in data]

#Create a dataframe (note - this dataframe still has columns that have nested attributes/features)
df_business = pd.DataFrame(data)
df_business.to_csv('business.csv')<jupyter_output><empty_output><jupyter_text>Checkin dataset<jupyter_code>#Open and read the json file
with open('./yelp_dataset/checkin.json', 'rb') as f:
    data = f.readlines()

#Convert each row of the Json file into a dictionary
data = [eval(x.decode().rstrip().replace('null','None')) for x in data]

#Create a dataframe (note - this dataframe still has columns that have nested attributes/features)
df_checkin = pd.DataFrame(data)
df_checkin.to_csv('checkin.csv')<jupyter_output><empty_output><jupyter_text>Tip dataset<jupyter_code>#Open and read the json file
with open('./yelp_dataset/tip.json', 'rb') as f:
    data = f.readlines()

#Convert each row of the Json file into a dictionary
data = [eval(x.decode().rstrip().replace('null','None')) for x in data]

#Create a dataframe (note - this dataframe still has columns that have nested attributes/features)
df_tip = pd.DataFrame(data)
df_tip.to_csv('tip.csv')<jupyter_output><empty_output><jupyter_text>User dataset<jupyter_code>#Open and read the json file
with open('./yelp_dataset/user.json', 'rb') as f:
    data = f.readlines()

#Convert each row of the Json file into a dictionary
data = [eval(x.decode().rstrip().replace('null','None')) for x in data]

#Create a dataframe (note - this dataframe still has columns that have nested attributes/features)
df_user = pd.DataFrame(data)
df_user.to_csv('user.csv')<jupyter_output><empty_output><jupyter_text>Review dataset<jupyter_code>#Open and read the json file
with open('./yelp_dataset/review.json', 'rb') as f:
    data = f.readlines()

#Convert each row of the Json file into a dictionary
data = [eval(x.decode().rstrip().replace('null','None')) for x in data]

#Create a dataframe (note - this dataframe still has columns that have nested attributes/features)
df_review = pd.DataFrame(data)
df_review.to_csv('review.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10096700796359499, 'fraction_numerical': 0.001422070534698521, 'mean_word_length': 5.586142322097379, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '444763', 'license': 'Xnet', 'n_tokens_mistral': 1049, 'n_tokens_neox': 1021, 'n_words': 396, 'path': 'Data_extraction_json_to_csv.ipynb', 'repo_name': 'RFM322/capstone_wip', 'size': '3531'}