<jupyter_start><jupyter_text>The lifecycle outlines the major stages that projects typically execute. There are six stages to build a modelBusiness Understanding Data UnderstandingData PreparationEvaluationModeling Deployment<jupyter_code>#Now I am Going to import nessesary module 
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
%matplotlib inline
from sklearn import tree
from sklearn import metrics

cale_data=pd.read_csv("calendar.csv")
list_data=pd.read_csv("listings.csv")
revi_data=pd.read_csv("reviews.csv")
#Look at the head of calendar datasets
cale_data.head(20)
#find missing values in calendar dataset
cale_data.isnull().sum()
#to find % missing values in each columns
cale_data.isnull().sum()/cale_data.isnull().shape[0]
cale_data.shape
# cale_data.describe()<jupyter_output><empty_output><jupyter_text>Q1.My First Question is that how the price varies with the Date Not only available days are stored in data, it seems to be stored not available days.If the available values are f, the price values seems to be NaN.The price values are stored as object, not integer. This is caused the value stored like $xx.xx, and it is necessary to transform this column.<jupyter_code># Now Start to cleaning the calender datasets
def transformation(df):
    '''input:That has to be clean
    i am writing this function to convert the date object to data and converts the price 
    columns object to flot vlaues
    output: will be a clean dataframe'''
    df.dropna(inplace=True)
    df['date']=pd.to_datetime(df['date'])
    
    df['price']=df['price'].str.replace('$','')
    df['price']=df['price'].str.replace(",",'')
    df['price']=df['price'].astype(float)
    return df
transformation(cale_data)
#let's aggrigate price based on the date for analys this question
cale_data=cale_data.groupby("date")['price'].mean().reset_index()
cale_data
#lets see how the price very over a period of time by plotting a line chart
plt.figure(figsize=(15,8))
plt.plot(cale_data['date'],cale_data['price'])
plt.title("Average price of a listing by date")
plt.xlabel("Average")
plt.ylabel("Average price")
plt.show()
#for the same let's try to take  out the weekdays name from the data and the plot the price chart the weekdays
cale_data["day of the week"]=cale_data["date"].dt.day_name()
plt.figure(figsize=(15,8))
sns.boxplot(x='day of the week', y='price',data=cale_data ,palette="Blues", width=0.6)
plt.show()<jupyter_output><empty_output><jupyter_text>Now we can understand that From the above visualisation,we figured out that the prices of the listings were maximum from July to September and were minimum from January to March  How long is the period available for lending by rooms<jupyter_code>#Now we are going to analys listing dataset
#fistly look at the head of the data
list_data.head()
#Now find the missing values in the listing dataset
list_data.isnull().sum()
# if I want to see the % missing values in each column, i have use below line
list_data.isnull().sum()/list_data.isnull().shape[0]
#The Shape of listing data or no of rows and column in Listing 
list_data.shape
#look at the number of id and row in the table
print("Num of listings: ", list_data.id.count())
print("Num of rows: ", list_data.shape[0])<jupyter_output>Num of listings:  3818
Num of rows:  3818<jupyter_text>From the above statement we can see that every row represents uniqe listings<jupyter_code>#now describe the data with 8 most important operations
list_data['review_scores_rating'].describe().reset_index()
#Lets see review_per_month
plt.figure(figsize=(15,8))
plt.hist(list_data['reviews_per_month'].dropna().values,bins=80,color='b')
plt.grid()<jupyter_output><empty_output><jupyter_text>from the above figure we did see the review per month<jupyter_code>#lets see the histogram of review_scores_rating
plt.figure(figsize=(15,8))
plt.hist(list_data['review_scores_rating'].dropna().values,bins=80,color='b')
plt.grid()<jupyter_output><empty_output><jupyter_text>This is very right skewed distribution.The 75% or more values are 90 points. And the most common thing is 100 points.I can say the low score listings are minolity.<jupyter_code>#now we are going to estemate outliers becouse values are very large
list_data = list_data[list_data['maximum_nights'] <= 1500]

plt.figure(figsize=(15, 8))
plt.hist(list_data.maximum_nights, bins=100, color='b')
plt.xlabel('maximum nights')
plt.ylabel('listings count')
plt.grid()<jupyter_output><empty_output><jupyter_text>This was shown when investigating listing data. There were two groups in listings. It's a listing available at spots with a maximum nights less than a week and a listing of a sense of renting available for up to three years.<jupyter_code>plt.figure(figsize=(15, 8))
plt.hist(list_data.price, bins=100, color='r')
plt.grid()<jupyter_output><empty_output><jupyter_text>This is long tail distribution.Almost values are from 0 to 200.<jupyter_code>revi_data.head()
revi_data.isnull().sum()
revi_data.isnull().sum()/revi_data.isnull().shape[0]
revi_data.shape
list_data.head()
list_data.columns
def clean_data(df):
    """
    Input: the dataframe that has to be cleaned
    This function treats the missing values and treats the column suitably
    Output:A clean dataframe
    
    """
    
    df=df.copy()
    df=df.loc[:,['host_response_time','host_response_rate','host_acceptance_rate','host_is_superhost','neighbourhood_cleansed','property_type','room_type','accommodates','bathrooms','bedrooms','beds','bed_type','amenities','price','weekly_price','monthly_price','security_deposit','cleaning_fee','guests_included','extra_people','minimum_nights','maximum_nights','review_scores_rating','review_scores_accuracy','review_scores_cleanliness','review_scores_checkin','review_scores_communication','review_scores_location','review_scores_value','instant_bookable','cancellation_policy']]
  

    #Lets remove the % sign from the host_response_rate and host_acceptance_rate columns
    df['host_response_rate']=df['host_response_rate'].str.replace('%','')
    df['host_acceptance_rate']=df['host_acceptance_rate'].str.replace('%','')
    
    
    #Converting host_response_rate and host_acceptance rate to float type
    df['host_response_rate']=df['host_response_rate'].astype('float64')/100
    df['host_acceptance_rate']=df['host_acceptance_rate'].astype('float64')/100
   

    #Filling the missing values with 0
    df['host_response_rate'].fillna(0,inplace=True)
    df['host_acceptance_rate'].fillna(0,inplace=True)
    
    
    #Converting the columns with binomial values 
    #Let us  replace f with 0 and t with 1 in host_is_superhost column
    df['host_is_superhost']=df['host_is_superhost'].str.replace('f','0')
    df['host_is_superhost']=df['host_is_superhost'].str.replace('t','1')
    df['instant_bookable']=df['instant_bookable'].str.replace('f','0')
    df['instant_bookable']=df['instant_bookable'].str.replace('t','1')
    
    
    #Converting these two coumns as well to float and converting it from percentage to number
    df['host_is_superhost']=df['host_is_superhost'].astype('float64')
    df['instant_bookable']=df['instant_bookable'].astype('float64')
    
    
    #Filling the missing values with 0
    df['host_is_superhost'].fillna(0,inplace=True)
    df['instant_bookable'].fillna(0,inplace=True)
    
    
    #We can see that these all columns are strings and have $ sign.We need to remove the $ sign and convert the type to float
    df['price']=df['price'].str.replace('$','')
    df['weekly_price']=df['weekly_price'].str.replace('$','')
    df['monthly_price']=df['monthly_price'].str.replace('$','')
    df['security_deposit']=df['security_deposit'].str.replace('$','')
    df['cleaning_fee']=df['cleaning_fee'].str.replace('$','')
    df['extra_people']=df['extra_people'].str.replace('$','')
    
    
    #Also,removing the commas as well
    df['price']=df['price'].str.replace(',','')
    df['weekly_price']=df['weekly_price'].str.replace(',','')
    df['monthly_price']=df['monthly_price'].str.replace(',','')
    df['security_deposit']=df['security_deposit'].str.replace(',','')
    df['cleaning_fee']=df['cleaning_fee'].str.replace(',','')
    df['extra_people']=df['extra_people'].str.replace(',','')
    
    
    #Converting to float64
    df['price']=df['price'].astype('float64')
    df['weekly_price']=df['weekly_price'].astype('float64')
    df['monthly_price']=df['monthly_price'].astype('float64')
    df['security_deposit']=df['security_deposit'].astype('float64')
    df['cleaning_fee']=df['cleaning_fee'].astype('float64')
    df['extra_people']=df['extra_people'].astype('float64')
    
    #Replacing the missing values with 0
    df['price'].fillna(0,inplace=True)
    df['weekly_price'].fillna(0,inplace=True)
    df['monthly_price'].fillna(0,inplace=True)
    df['security_deposit'].fillna(0,inplace=True)
    df['cleaning_fee'].fillna(0,inplace=True)
    df['extra_people'].fillna(0,inplace=True)
    
    
    #Let us clean  the ameneties column
    df['amenities']=df['amenities'].str.replace("{","")
    df['amenities']=df['amenities'].str.replace("}","")
    df['amenities']=df['amenities'].str.replace(" ","")
    df['amenities']=df['amenities'].str.replace('"','')
    df['amenities']=df['amenities'].str.replace(",","  ")
    df['amenities']
    df['amenities_length']=df['amenities'].str.len()
    df=df.drop(['amenities'],axis=1)
    
    
    #Converting these categories to integer values
    df['host_response_time'].replace('within an hour',1, inplace=True)
    df['host_response_time'].replace('within a few hours',2,inplace=True)
    df['host_response_time'].replace('within a day',3,inplace=True)
    df['host_response_time'].replace('a few days or more',4,inplace=True)
    
    
    #Filling the missing values of this column as well
    df['host_response_time'].fillna(method='ffill',inplace=True)
    
    
    #Let us also treat missing values of some other columns
    df['beds'].fillna(method='ffill',inplace=True)
    df['bathrooms'].fillna(method='ffill',inplace=True)
    df['bedrooms'].fillna(method='ffill',inplace=True)
    return df
listing_ratings=clean_data(list_data)
listing_ratings
listing_ratings['property_type'].unique()
def assign1(val):
    if val in ['House', 'Cabin', 'Condominium','Loft', 'Boat','Tent']:
        return val
    else:
        return 'other';
listing_ratings['property_type']=listing_ratings['property_type'].apply(assign1)
def assign2(val):
    if val in ['red bed']:
        return val
    return 'other'
listing_ratings['property_type']=listing_ratings['property_type'].apply(assign2)
print("sample 1: ", revi_data.comments.values[0], "\n")
print("sample 2: ", revi_data.comments.values[3])
# convert date column's data type to date from object
review_df= revi_data.copy(deep=True)
review_df.date = pd.to_datetime(review_df.date)

review_df = review_df.groupby('date')['id'].count().reset_index()

# plot avg listings prices over time.
plt.figure(figsize=(15, 8))
plt.plot(review_df.date, review_df.id, color='b', linewidth=0.9)
plt.title("Number of reviews by date")
plt.xlabel('date')
plt.ylabel('number of reviews')
plt.grid()<jupyter_output><empty_output><jupyter_text>It is little noisy, but we can see an increase in the number of Airbnb users. (and the date range is wide than calendar data)And I realize it seems to have a peak at about the same time of each year.So, let's use moving averages to smooth the graph.<jupyter_code># create rolling mean column
review_df["rolling_mean_30"] = review_df.id.rolling(window=30).mean()

# plot avg listings prices over time.
plt.figure(figsize=(15, 8))
plt.plot(review_df.date, review_df.rolling_mean_30, color='b', linewidth=2.0)
plt.title("Number of reviews by date")
plt.xlabel('date')
plt.ylabel('number of reviews')
plt.grid()<jupyter_output><empty_output><jupyter_text>I tried thirty days (about 1 month) window.The graph became smooth and the trend became clear, and my belief that the peaks were in the same place became stronger.<jupyter_code>listings_df["min_max_night_diff"] = listings_df.maximum_nights - listings_df.minimum_nights

plt.figure(figsize=(15, 8))
plt.plot(listings_df.maximum_nights, listings_df.minimum_nights, color='b', marker='o', linewidth=0, alpha=0.25)
plt.xlabel('maximum nights')
plt.ylabel('minimum nights')
plt.grid()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14012994304965107, 'fraction_numerical': 0.011630705061361995, 'mean_word_length': 5.980963045912654, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 40, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '650232', 'license': 'FTL', 'n_tokens_mistral': 4265, 'n_tokens_neox': 4029, 'n_words': 979, 'path': '.ipynb_checkpoints/Untitled-checkpoint.ipynb', 'repo_name': 'amiruddinsaifi01/Project1', 'size': '12467'}