<jupyter_start><jupyter_text>Know Your Seattle - Analysis of Seattle Airbnb dataThe goal of this project is to analyse Airbnb open source Seattle dataset *https://www.kaggle.com/airbnb/seattle* and extract insights out of it using Cross Industry Standard Process for Data Mining or ***CRISP-DM*** methodology.CRISP-DM has following steps:* Business Understanding* Data Understanding* Prepare Data* Data Modeling* Evaluation Step1: Business UnderstandingAny accommodation domain always involves two parties, an owner and a tenant/visitor.As a visitor/tenant we may have following questions in my mind:* Which is the busiest season in Seattle ?* Does the accommodation rate varies with the season ?As an owner, we may be considering:* What are the parameters on which price of an accommodation depends ?So  in this project, we will try to cover both the perspective by answering the above questions.<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score, mean_squared_error
from sklearn.ensemble import RandomForestRegressor

import AllTogether as t
import seaborn as sns
from datetime import datetime
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Step2: Data Understanding<jupyter_code># read calendar data
df_calendar = pd.read_csv('./calendar.csv')
df_calendar.head()
# basic analysis
print('columns: \n', df_calendar.columns)
print('total rows: ',df_calendar.shape[0])
print('total columns: ',df_calendar.shape[1])
print('column types: \n', df_calendar.dtypes)<jupyter_output>columns: 
 Index(['listing_id', 'date', 'available', 'price'], dtype='object')
total rows:  1393570
total columns:  4
column types: 
 listing_id     int64
date          object
available     object
price         object
dtype: object<jupyter_text>***calendar*** dataset has four columns *Id, Date, Available & Price*. Total number of listings/accommodations are 3818 and each of them includes availability info (true/false) for complete 365 days. The most expensive listing stands at `$1650` and least expensive is at `$10`. The average cost of listing is `$138` and the standard deviation is `$105` in Seattle.<jupyter_code>df_listings = pd.read_csv('./listings.csv')
df_listings.head()
df_listings['review_scores_rating'].describe()<jupyter_output><empty_output><jupyter_text>***listings*** dataset has a total of 92 columns storing all the info related to 3818 accommodations such as owner, features, location, neighborhood, reviews etc. The average current price is `$128`, average security deposit is `$288`, average cleaning fee is `$68`and average extra people cost is `$11` in Seattle. There are hosts as old as since 10 Nov 2008 to as new as 03 Jan 2016. Around 778 or 20% owners got ***super host*** rating. Overall, average rating for an owner in Seattle is 94 out of 100 which is quite good.  Step3: Prepare Data<jupyter_code>def process_price(df, col):
    """Converts the price values that are string with special characters to numbers."""
    
    df[col] = df[col].replace('[$,]','', regex=True).astype(float)
# remove unwanted characters from price and convert it to numeric
process_price(df_calendar, 'price')
df_calendar['price'].head()
df_calendar['price'].describe()
def process_date(df, col):
    """Converts the date that are string to date_time values."""
    
    df[col] = pd.to_datetime(df[col], format='%Y-%m-%d')
# convert string dates to date_time values
process_date(df_calendar, 'date')
df_calendar.head()
# determine seasons, ref: https://www.timeanddate.com/calendar/aboutseasons.html
seasons = {
    1: 'winter',
    2: 'winter',
    3: 'spring',
    4: 'spring',
    5: 'spring',
    6: 'summer',
    7: 'summer',
    8: 'summer',
    9: 'fall',
    10: 'fall',
    11: 'fall',
    12: 'winter'
}
# add season info in dataframe
df_calendar['date_season'] = df_calendar['date'].dt.month.apply(lambda x: seasons[x])
df_calendar['date_season'].value_counts()<jupyter_output><empty_output><jupyter_text>***NOTE:*** Here we see number of records in winter is less than summer. This difference has an impact in our final result.<jupyter_code>df_calendar['available'].value_counts()
def process_boolean(df, col):
    """Converts boolean true to 1 and boolean false to 0."""
    
    df[col] = df[col].replace({'t': 1, 'f': 0})
# convert boolean string to boolean int
process_boolean(df_calendar, 'available')
df_calendar['available'].value_counts()
def plot_data(df):
    """Plots the data in dataframe as a bar graph."""
    
    df.plot(kind = 'barh')
# find availability per season, the lower the busier
availability_per_season = df_calendar.groupby(df_calendar['date_season'])['available'].sum().sort_values()
# find price per season, more pricy means more demanding
price_per_season = df_calendar.groupby(df_calendar['date_season'])['price'].mean().sort_values(ascending = False)<jupyter_output><empty_output><jupyter_text>Data preprocessing of listings dataset<jupyter_code># identify missing values of more than 75%
set(df_listings.columns[df_listings.isnull().mean() > 0.95])<jupyter_output><empty_output><jupyter_text>I have chosen 95% as the threshold for missing count as such columns will not have any weightage in our prediction model<jupyter_code># delete the most missing values columns
df_listings = df_listings.drop(columns = ['license', 'square_feet'])
df_listings.columns<jupyter_output><empty_output><jupyter_text>The following columns are decided as unimportant because they would not help in prediction modeling:* unique & random id's and url's etc. as strings* string data such as notes, summary, description, neighborhood overview etc.* redundant data such as city, state, country code etc.* complex parameters such as neighborhood or amenities as they would complicate our model* skewed data such as smart_location, has_availability etc.<jupyter_code>df_listings['has_availability'].value_counts()
# columns that would not help in our modeling
unimportant_columns = ['id', 'listing_url', 'scrape_id', 'last_scraped', 'name', 'summary', 'space', 'description',
                     'experiences_offered', 'neighborhood_overview', 'notes', 'transit', 'thumbnail_url', 
                      'medium_url', 'picture_url', 'xl_picture_url', 'host_id', 'host_url', 'host_name',
                      'host_location', 'host_about', 'host_thumbnail_url', 'host_picture_url', 'host_neighbourhood',
                      'host_verifications', 'street', 'neighbourhood', 'city', 'state', 'zipcode', 'country_code', 
                      'neighbourhood_cleansed', 'neighbourhood_group_cleansed', 'amenities','host_listings_count', 
                       'market', 'smart_location', 'country', 'calendar_updated', 'has_availability', 
                       'calendar_last_scraped', 'first_review', 'last_review', 'jurisdiction_names', 
                       'calculated_host_listings_count']
df_listings = df_listings.drop(columns = unimportant_columns)
df_listings.head()
print(df_listings.shape[0])
print(df_listings.shape[1])
# identify price columns, they have special characters
price_columns = ['price', 'weekly_price', 'monthly_price', 'security_deposit', 'cleaning_fee', 'extra_people']
# remove unwanted characters from price and convert it to numeric
for price in price_columns:
    process_price(df_listings, price)
for price in price_columns:
    print('col: {}, average: {}'.format(price, df_listings[price].mean()))
boolean_columns = ['host_is_superhost', 'host_has_profile_pic', 'host_identity_verified', 'is_location_exact',
                  'instant_bookable', 'require_guest_profile_picture', 'require_guest_phone_verification']
df_listings['host_is_superhost'].value_counts()
df_listings['requires_license'].value_counts()
# remove column whose data is skewed
df_listings = df_listings.drop(columns = ['requires_license'])
# convert boolean string to boolean int
for boolean_column in boolean_columns:
    process_boolean(df_listings, boolean_column)
df_listings['host_acceptance_rate'].value_counts()
# remove column whose data is skewed
df_listings = df_listings.drop(columns = ['host_acceptance_rate'])
def process_percentage(df, col):
    """Converts percentage values to numbers."""
    
    df[col] = df[col].replace('%', '', regex=True).astype(float)
# remove unwanted characters from percentage column and convert it to numeric
process_percentage(df_listings, 'host_response_rate')
# convert string dates to date_time values
process_date(df_listings, 'host_since')
df_listings['host_since'].describe()
df_listings['host_since'].tail()
def convert_date_to_number(df, col, first_day):
    """Converts date to number of days since first day."""
    
    df[col] = df[col].apply(lambda date: (first_day - date).days)
# find first day of the data, then host_since (in days) = (first_day - host_since).days
first_day = pd.to_datetime('2016-01-04')
print(first_day)
# convert date_time values to number of days numeric
convert_date_to_number(df_listings, 'host_since', first_day)
df_listings['host_since'].tail()
df_listings.columns<jupyter_output><empty_output><jupyter_text>Drop any row that does not have a value for response column ***price***<jupyter_code># drop any row where price is NAN
df_listings = df_listings.dropna(subset=['price'], axis=0)
# identify numeric columns
numeric_columns = df_listings.select_dtypes(include=['float', 'int']).columns
numeric_columns<jupyter_output><empty_output><jupyter_text>These numeric columns are important for our prediction. As sklearn's fit() method does not work with NANs, we have to impute the NANs. As these are numbers, mean of the colums is appropriate for imputing NANs.<jupyter_code># fill missing numeric column with mean of the column
for col in numeric_columns:
        df_listings[col].fillna((df_listings[col].mean()), inplace=True)
df_listings.head()
# identify categorical columns
categorical_columns = df_listings.select_dtypes(include=['object']).columns
categorical_columns<jupyter_output><empty_output><jupyter_text>As categorical values are important for our modeling step, instead of dropping we will encode them with 1's and 0's. One advantage of this is with linear modeling, it's easy to interpret weights on these categorical values. Also, since the number of new dummy-columns created is far less than the number of rows, thus, we are safe from its scalability issues. We must also be aware that NANs in these columns would be encoded with 0 and this may impact our result.<jupyter_code># fill missing categorical columns with dummy data
for col in  categorical_columns:
        df_listings = pd.concat([df_listings.drop(col, axis=1), pd.get_dummies(df_listings[col], prefix=col, prefix_sep='_', drop_first=True)], axis=1)
df_listings.head()<jupyter_output><empty_output><jupyter_text>Step4: Data Modeling<jupyter_code>y = df_listings['price']
# remove the response column
df_listings = df_listings.drop(['price'], axis=1)
X = df_listings
#split the data into train and test
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.3, random_state=42)
print('Training sample: {}'.format(X_train.shape[0]))
print('Testing sample: {}'.format(X_test.shape[0]))

# instantiate model
rf_model = RandomForestRegressor()
# fit training data
rf_model.fit(X_train, y_train)
# predict price on training and test data
y_test_preds = rf_model.predict(X_test)
y_train_preds = rf_model.predict(X_train)

train_score = r2_score(y_train, y_train_preds)
print('train_score: {}'.format(train_score))
test_score = r2_score(y_test, y_test_preds)
print('test_score: {}'.format(test_score))

feature_importances = pd.Series(rf_model.feature_importances_, index=df_listings.columns)<jupyter_output>Training sample: 2672
Testing sample: 1146
train_score: 0.9251969660428926
test_score: 0.6700559978786749<jupyter_text>Step5: Evaluation<jupyter_code>plot_data(availability_per_season)<jupyter_output><empty_output><jupyter_text>The plot above answers our first question. Summer shows a low count as most of the tourists prefer this season leading to low availability. Surprisingly, winter was the lowest, but this can also be attributed to the reason that number of days during in winter (90) in this dataset were lesser than the number of days in summer (92). Thus, considering the case of equal number of days, ***we can conclude that the Summer is the busiest season***.<jupyter_code>plot_data(price_per_season)<jupyter_output><empty_output><jupyter_text>The plot above answers our second question. We can clearly conclude that ***price does vary as per the season*** and busy season (as summer) means high prices too.<jupyter_code>plot_data(feature_importances.nlargest(10))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10399249941401673, 'fraction_numerical': 0.013594812094694897, 'mean_word_length': 5.881720430107527, 'pattern_counts': {'":': 0, '<': 59, '<?xml version=': 0, '>': 60, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '635898', 'license': 'CC0-1.0', 'n_tokens_mistral': 3972, 'n_tokens_neox': 3768, 'n_words': 1229, 'path': 'seattle_airbnb_analysis.ipynb', 'repo_name': 'govindsinghnegi/DSND-T2-Project1', 'size': '12799'}