<jupyter_start><jupyter_text>Common Functions Importiamo e puliamo i dati in un unico notebook.Comodo per non dover tutte le volte rifare la stessa cosa- Import dei dati  - listings  - reviews  - calendar- Pulizia dei dati- Salvataggio finale<jupyter_code>import pandas as pd
import numpy as np
import os
#Check Dataframe Utility function
def check_df(dataframe, sample=False):
    
    print(f"Dataframe Shape: {dataframe.shape} with rows: {dataframe.shape[0]} and columns: {dataframe.shape[1]}")
    print(f"\nDF Columns: \n{list(dataframe.columns)}")
    if sample == True:
        print(f"\nData:\n{dataframe.head(5)}")
    
    return None<jupyter_output><empty_output><jupyter_text>Import dei dati<jupyter_code>folder_path = os.path.abspath("..")
data_dir = 'dataset/rome-airbnb'

#listings
filename_listings = 'listings.csv'
filepath_listings = os.path.join(folder_path, data_dir, filename_listings)

#reviews
filename_reviews = 'reviews.csv'
filepath_reviews = os.path.join(folder_path, data_dir, filename_reviews)

#calendar
filename_calendar = 'calendar.csv'
filepath_calendar = os.path.join(folder_path, data_dir, filename_calendar)<jupyter_output><empty_output><jupyter_text>Import Listings<jupyter_code>%%time
#Import listings
listings_raw = pd.read_csv(filepath_listings)
check_df(listings_raw)<jupyter_output>Dataframe Shape: (3818, 92) with rows: 3818 and columns: 92

DF Columns: 
['id', 'listing_url', 'scrape_id', 'last_scraped', 'name', 'summary', 'space', 'description', 'experiences_offered', 'neighborhood_overview', 'notes', 'transit', 'thumbnail_url', 'medium_url', 'picture_url', 'xl_picture_url', 'host_id', 'host_url', 'host_name', 'host_since', 'host_location', 'host_about', 'host_response_time', 'host_response_rate', 'host_acceptance_rate', 'host_is_superhost', 'host_thumbnail_url', 'host_picture_url', 'host_neighbourhood', 'host_listings_count', 'host_total_listings_count', 'host_verifications', 'host_has_profile_pic', 'host_identity_verified', 'street', 'neighbourhood', 'neighbourhood_cleansed', 'neighbourhood_group_cleansed', 'city', 'state', 'zipcode', 'market', 'smart_location', 'country_code', 'country', 'latitude', 'longitude', 'is_location_exact', 'property_type', 'room_type', 'accommodates', 'bathrooms', 'bedrooms', 'beds', 'bed_type', 'amenities', 'square_feet', 'price', [...]<jupyter_text>Import Reviews<jupyter_code>%time
#Import reviews
sample_dim = 100000

reviews_raw = pd.read_csv(filepath_reviews).head(sample_dim)
check_df(reviews_raw)<jupyter_output>CPU times: user 2 µs, sys: 0 ns, total: 2 µs
Wall time: 4.29 µs
Dataframe Shape: (84849, 6) with rows: 84849 and columns: 6

DF Columns: 
['listing_id', 'id', 'date', 'reviewer_id', 'reviewer_name', 'comments']<jupyter_text>Import Calendar<jupyter_code>%time
calendar_raw = pd.read_csv(filepath_calendar).head(sample_dim)
check_df(calendar_raw)<jupyter_output>CPU times: user 3 µs, sys: 0 ns, total: 3 µs
Wall time: 4.77 µs
Dataframe Shape: (100000, 4) with rows: 100000 and columns: 4

DF Columns: 
['listing_id', 'date', 'available', 'price']<jupyter_text>Pulizia Dati  Pulizia Listings<jupyter_code># Selezioniamo solo alcune delle colonne
listings = listings_raw[[
    'id','name','longitude','latitude',
    'listing_url',
    'instant_bookable',
    'host_response_time',
    'review_scores_rating',
    'property_type',
    'room_type','accommodates',
    'bathrooms','bedrooms','beds','reviews_per_month','amenities',
    'number_of_reviews',
    'price'
  ]]
listings.head()<jupyter_output><empty_output><jupyter_text>Price da stringa a numero reale<jupyter_code>@np.vectorize
def remove_dollar(label: str):
    return float(label.replace('$','').replace(',',''))

#listings['price'] = remove_dollar(listings['price'])
listings = listings.assign(price = remove_dollar(listings.price))
listings[['price']]<jupyter_output><empty_output><jupyter_text>Pulizia Reviews<jupyter_code>#Date to datetime
reviews = reviews_raw.assign(date = pd.to_datetime(reviews_raw['date']))<jupyter_output><empty_output><jupyter_text>Creazione delle variabili mese e anno<jupyter_code>%timeit
reviews['year'] = reviews['date'].dt.year
reviews['month'] = reviews['date'].dt.month
reviews = reviews.sort_values(['year', 'month'], ascending=False)
reviews.shape<jupyter_output><empty_output><jupyter_text>Pulizia Calendar<jupyter_code>calendar = calendar_raw.assign(date = pd.to_datetime(calendar_raw['date']))<jupyter_output><empty_output><jupyter_text>Preparazione del prezzo<jupyter_code>calendar = calendar.assign(
        price          = pd.to_numeric(calendar.price.str.replace('$','').str.replace(',','')),
        # adjusted_price = pd.to_numeric(calendar.adjusted_price.str.replace('$','').str.replace(',','')),
    )
%timeit
calendar['year'] = pd.DatetimeIndex(calendar['date']).year
calendar['month'] = pd.DatetimeIndex(calendar['date']).month
calendar = calendar.sort_values(['year', 'month'], ascending=False)
calendar.shape<jupyter_output><empty_output><jupyter_text>Mappatura dei valori logici<jupyter_code>calendar['available'] = calendar.available.map({
    't': True,
    'f': False
})<jupyter_output><empty_output><jupyter_text>Final Export Export finale dei dati e salvataggio<jupyter_code>df_listing = listings.copy()
df_review = reviews.copy()
df_calendar = calendar.copy()

print("\n---Cleaning and Parsing of the data Completed---\n")
print(f"Listing parsed and saved with name: df_listing")
print(f"Review parsed and saved with name: df_review")
print(f"Calendar parsed and saved with name: df_calendar")<jupyter_output>---Cleaning and Parsing of the data Completed---

Listing parsed and saved with name: df_listing
Review parsed and saved with name: df_review
Calendar parsed and saved with name: df_calendar
{'fraction_non_alphanumeric': 0.14412842435875065, 'fraction_numerical': 0.010294887454196475, 'mean_word_length': 7.632530120481928, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '109998', 'license': 'MIT', 'n_tokens_mistral': 2075, 'n_tokens_neox': 1987, 'n_words': 395, 'path': 'PercorsoDati/Lab3-4/notebooks/lab-1-data.ipynb', 'repo_name': 'nick87ds/MaterialeSerate', 'size': '5731'}