<jupyter_start><jupyter_text>Database Preparation  In this notebook we will prepare our data for our search function to use. Currently we have data stored in two different csv files. * cars.csv * newcars.csvIt can be computationally expensive to produce analysis results from multiple data-sources for incomming stream of requests.So we will prepare our data and save it in an easily searchable structure<jupyter_code># Import the needed modules...
import pandas as pd
from collections import defaultdict
from os import getcwd<jupyter_output><empty_output><jupyter_text>Define Paths to data files.<jupyter_code>PATH_CARS   = f"{getcwd()}/datastore/cars.csv"
PATH_NEWCARS  = f"{getcwd()}/datastore/newcars.csv"<jupyter_output><empty_output><jupyter_text>Data Engineering* Get data in dataframes.* Convert data to a single dictionary.<jupyter_code>"""
    Read data from cars.csv
"""
df_cars            = pd.read_csv(PATH_CARS)
cars_table_columns = df_cars.columns.tolist()
print(f"COLUMNS : {cars_table_columns}")
"""
    Read data from newcars.csv
"""
df_newcars          = pd.read_csv(PATH_NEWCARS)
newcars_table_columns = df_newcars.columns.tolist()
print(f"COLUMNS : {newcars_table_columns}")<jupyter_output>COLUMNS : ['id', 'car_name', 'yr_mfr', 'fuel_type', 'kms_run', 'sale_price', 'city', 'times_viewed', 'body_type', 'transmission', 'variant', 'assured_buy', 'registered_city', 'registered_state', 'is_hot', 'rto', 'source', 'make', 'model', 'car_availability', 'total_owners', 'broker_quote', 'original_price', 'car_rating', 'ad_created_on', 'fitness_certificate', 'emi_starts_from', 'booking_down_pymnt', 'reserved', 'warranty_avail']<jupyter_text>* "id" is a common column in both the tables so we will use it as a primary    search-key*  A user will always search a car by its "car_name" so we will create a        GLOBAL SECONDARY INDEX to be able to perform search our datastore<jupyter_code>print(f"It is {pd.Series(df_cars['id']).is_unique}  that the column 'id' has unique values for all entries in cars dataframe.")
print(f"It is {pd.Series(df_newcars['id']).is_unique}  that the column 'id' has unique values for all entries in newcars dataframe.")
# Sort cars dataframe on the basis of id as id is unique for all entries...
df_cars_sorted = df_cars.sort_values(by=['id'])

# Sort newcars dataframe on the basis of id as id is unique for all entries...
df_newcars_sorted  = df_newcars.sort_values(by=['id'])
# from cars dataframe...
carsIds    = df_cars_sorted["id"].tolist()
carnamess = df_cars_sorted["car_name"].tolist()
fueltypes = [fueltype.split("|") for fueltype in df_cars["fuel_type"].tolist()]

# from newcars dataframe...
car_rating  = df_newcars_sorted["car_rating"].tolist()
original_price = df_newcars_sorted["original_price"].tolist()
carsDict             = {}
global_secondaryIndex = {}
for idx, id in enumerate(carsIds):
    carsDict[id] = {
        "fueltype" : fueltypes[idx],
        "newcars" : {
            "car_rating" : car_rating[idx], 
            "original_price" : original_price[idx]
        }
    }
    
    global_secondaryIndex[carnamess[idx]] = id
# delete veriables which are no longer in use while holding large amount of data.
del carsIds
del carnamess
del fueltypes
del car_rating
del original_price

import json
print("[INFO] Writing car Data into the disk...")
with open('dataStore/dataFinal.json', 'w') as fp:
    json.dump(carsDict, fp, sort_keys=True, indent=4)
print("[INFO] Writing Global Secondary Index Data into the disk...")
with open('dataStore/dataFinal_GIS.json', 'w') as fp:
    json.dump(global_secondaryIndex, fp, sort_keys=True, indent=4)<jupyter_output>[INFO] Writing car Data into the disk...
[INFO] Writing Global Secondary Index Data into the disk...
{'fraction_non_alphanumeric': 0.11692142665593994, 'fraction_numerical': 0.0005363368195226602, 'mean_word_length': 5.084828711256117, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '418931', 'license': 'CC0-1.0', 'n_tokens_mistral': 1256, 'n_tokens_neox': 1200, 'n_words': 362, 'path': 'DataPreparation.ipynb', 'repo_name': 'Gokul58/expert_session_one_Gokul-S-Kumar', 'size': '3729'}