<jupyter_start><jupyter_text>Project 1 - Analysing Seattle and Boston Airbnb datasets In this project we will try to answer this 4 folowing questions:1) When is there more (or less) availability in Boston and Seattle? Is there a difference between the two cities?2) When is it cheaper to stay in Boston or Seattle? Is there a difference between the two cities?3) Do neighborhoods influence prices? And the score rating?4) What features do the top rated (score rating +90) properties have in common? What is different from the other properties? In this notebook, we will answer 4) questions<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt

%matplotlib inline
# import data
df_boston_listings = pd.read_csv('./airbnb-boston/listings.csv')
df_seattle_listings = pd.read_csv('./airbnb-seattle/listings.csv')
# create def
def get_percents(df_top, df_all, column):
    '''
    INPUT:
    df_top - the pandas dataframe with the top rated properties (+90 score)
    df_all - the pandas dataframe with all properties
    column - the column you want to count values from
    
    
    OUTPUT:
    df_result - holds percentage of 'column' presents in 'df_top', percentage of 'column' presents in 'df_all' and the difference beetween these two percentages 
    '''
    df_top = df_top[['id',column]].groupby(column).count().reset_index()
    df_top['top'] = df_top['id']/df_boston_top.shape[0]
    df_top[column] = df_top[column].astype(str)
    
    df_all = df_all[['id',column]].groupby(column).count().reset_index()
    df_all['all']= df_all['id']/df_boston_values.shape[0]
    df_all[column] = df_all[column].astype(str)
    
    df_result = pd.merge(df_all, df_top, how='left', on=column)
    df_result['dif'] = (df_result['top']-df_result['all'])*100
    df_result = df_result[[column,'top','all', 'dif']].set_index(column)
    df_result=df_result.fillna(0)
    
    return df_result.sort_values(by='dif', ascending=False)
def plot(df):
    '''
    INPUT:
    df - the pandas dataframe you want to plot
    
    OUTPUT:
    a plot with the percentages in bars and the difference in line
    '''
    fig = plt.figure()
    ax = df[['top','all']].plot(figsize=(35,10), kind='bar')
    ax.set_ylim(max(max(df['top']),max(df['all']))*-1,max(max(df['top']),max(df['all'])))
    plt.xticks(rotation=0)
    ax2 = ax.twinx()
    ax2.plot(ax.get_xticks(), df[['dif']], color='red')
    ax2.set_ylim(max(abs(df['dif']))*-1,max(abs(df['dif'])))
    plt.show()<jupyter_output><empty_output><jupyter_text>Analyzing Boston data:<jupyter_code># looking into data
df_boston_listings.head(1)
# filter dataframe keeping the columns that are interesting
df_boston_values = df_boston_listings[['id','review_scores_rating','neighbourhood_cleansed','property_type','room_type','bathrooms','bedrooms','beds','bed_type','amenities','cancellation_policy']]
df_boston_values.describe(include='all')
# convert amenities list in quantity os amenities
df_boston_values['amenities'] = df_boston_values['amenities'].apply(lambda x: len(x.split(',')))
# create dataframe with top rated properties
df_boston_top = df_boston_values[df_boston_values['review_scores_rating']>=90]

# fills nan values with 'NA'
df_boston_top=df_boston_top.fillna('NA')
df_boston_values = df_boston_values.fillna('NA')
df_boston_values.dtypes
df_boston_top.head(1)<jupyter_output><empty_output><jupyter_text>Neighbourhood_cleansed analysis:<jupyter_code>nb = get_percents(df_boston_top, df_boston_values, 'neighbourhood_cleansed')
nb.head(2)
nb.tail(2)
plot(nb)<jupyter_output><empty_output><jupyter_text>- There is no big difference in the distribution of properties in the **neighborhoods** when we compare the top rated with the general. The biggest differences are in Jamaica Palin, which appears almost 2.8% (percentage points) among the top rated and in Fenway, which appears almost 2.4% less.  Property_type analysis:<jupyter_code>pt = get_percents(df_boston_top, df_boston_values, 'property_type')
pt.head(2)
pt.tail(2)
plot(pt)<jupyter_output><empty_output><jupyter_text>- There is no big difference in the distribution of the **properties types** when we compare the top rated with the general. The biggest differences are in House, which appears 2% (percentage points) more among the top rated and in Apartment, which appears almost 3.5% less.  Room_type analysis:<jupyter_code>rt = get_percents(df_boston_top, df_boston_values, 'room_type')
rt.head(2)
rt.tail(2)
plot(rt)<jupyter_output><empty_output><jupyter_text>- There is no big difference in the distribution of **room types** when we compare the top rated with the general.   Bathrooms analysis:<jupyter_code>bath = get_percents(df_boston_top, df_boston_values, 'bathrooms')
bath.head(2)
bath.tail(2)
plot(bath)<jupyter_output><empty_output><jupyter_text>- There is no big difference in the distribution of **bathroom quantity** when we compare the top rated with the general. The biggest differences are in properties with 2 bathrooms, which appears 1% (percentage point) more among the top rated.  Bedrooms analysis:<jupyter_code>bed = get_percents(df_boston_top, df_boston_values, 'bedrooms')
bed.head(2)
bed.tail(2)
plot(bed)<jupyter_output><empty_output><jupyter_text>- There is no big difference in the distribution of **beds quantity** when we compare the top rated with the general.   Bed_type analysis:<jupyter_code>bt = get_percents(df_boston_top, df_boston_values, 'bed_type')
bt.head(2)
bt.tail(2)
plot(bt)<jupyter_output><empty_output><jupyter_text>- There is no big difference in the distribution of the **bed types** when we compare the top rated with the general.   Amenities amount analysis:<jupyter_code>aa = get_percents(df_boston_top, df_boston_values, 'amenities')
aa.head(2)
aa.tail(2)
plot(aa)<jupyter_output><empty_output><jupyter_text>- There is no big difference in the distribution of the **amenities quantity** when we compare the top rated with the general. The biggest differences are in properties with 20 and 23 amenities, which appears 1.3% and 1% (percentage point) more among the top rated.  Cancellation_policy analysis:<jupyter_code>cp = get_percents(df_boston_top, df_boston_values, 'cancellation_policy')
cp.head(2)
cp.tail(2)
plot(cp)<jupyter_output><empty_output><jupyter_text>- There is some difference in the distribution of the **cancelllation policy** when we compare the top rated with the general. The biggest differences are in properties with moderate, which appears 4.3% (percentage points) more among the top rated and in properties with flexible, which appears 4.7% less.  Analyzing Seattle data:<jupyter_code># looking into data
df_seattle_listings.head(1)
# filter dataframe keeping the columns that are interesting
df_seattle_values = df_seattle_listings[['id','review_scores_rating','neighbourhood_cleansed','property_type','room_type','bathrooms','bedrooms','beds','bed_type','amenities','cancellation_policy',]]
df_seattle_values.describe(include='all')
# convert amenities list in quantity os amenities
df_seattle_values['amenities'] = df_seattle_values['amenities'].apply(lambda x: len(x.split(',')))
# create dataframe with top rated properties
df_seattle_top = df_seattle_values[df_seattle_values['review_scores_rating']>=90]

# fills nan values with 'NA'
df_seattle_top=df_seattle_top.fillna('NA')
df_seattle_values = df_seattle_values.fillna('NA')
df_seattle_values.dtypes
df_seattle_top.head(1)<jupyter_output><empty_output><jupyter_text>Neighbourhood_cleansed analysis:<jupyter_code>s_nb = get_percents(df_seattle_top, df_seattle_values, 'neighbourhood_cleansed')
s_nb.head(7)
s_nb.tail(2)
plot(s_nb)<jupyter_output><empty_output><jupyter_text>- There is no big difference in the distribution of properties in the **neighborhoods** when we compare the top rated with the general. The biggest difference is in Broadway, which appears 2% (percentage points) more among the top rated.  Property_type analysis:<jupyter_code>s_pt = get_percents(df_seattle_top, df_seattle_values, 'property_type')
s_pt.head(2)
s_pt.tail(2)
plot(s_pt)<jupyter_output><empty_output><jupyter_text>- There is a big difference in the distribution of the **properties types** when we compare the top rated with the general. The biggest differences are at House and Apartment, which appears 14% and 10% (percentage points) more among the top rated.  Room_type analysis:<jupyter_code>s_rt = get_percents(df_seattle_top, df_seattle_values, 'room_type')
s_rt.head(2)
s_rt.tail(2)
plot(s_rt)<jupyter_output><empty_output><jupyter_text>- There is a big difference in the distribution of the **room types** when we compare the top rated with the general. The biggest differences are at Entire home/apart and Private, which appears 18% and 8% (percentage points) more among the top rated.  Bathrooms analysis:<jupyter_code>s_bath = get_percents(df_seattle_top, df_seattle_values, 'bathrooms')
s_bath.head(2)
s_bath.tail(2)
plot(s_bath)<jupyter_output><empty_output><jupyter_text>- There is a big difference in the distribution of the **bathrooms** when we compare the top rated with the general. The biggest differences are athe properties with 1 bathroom, which appears 20% (percentage points) more among the top rated.  Bedrooms analysis:<jupyter_code>s_bed = get_percents(df_seattle_top, df_seattle_values, 'bedrooms')
s_bed.head(2)
s_bed.tail(2)
plot(s_bed)<jupyter_output><empty_output><jupyter_text>- There is a big difference in the distribution of the **bedrooms** when we compare the top rated with the general. The biggest differences are at properties with 1 bedroom, which appears 18% (percentage points) more among the top rated.  Bed_type analysis:<jupyter_code>s_bt = get_percents(df_seattle_top, df_seattle_values, 'bed_type')
s_bt.head(2)
s_bt.tail(2)
plot(s_bt)<jupyter_output><empty_output><jupyter_text>- There is a big difference in the distribution of the **bed types** when we compare the top rated with the general. The biggest differences are at Real bed, which appears 26% (percentage points) more among the top rated.  Amenities amount analysis:<jupyter_code>s_aa = get_percents(df_seattle_top, df_seattle_values, 'amenities')
s_aa.head(2)
s_aa.tail(2)
plot(s_aa)<jupyter_output><empty_output><jupyter_text>- There is no big difference in the distribution of the **amenities quantity** when we compare the top rated with the general. The biggest differences are at properties with 15 amenities, which appears 3% (percentage points) more among the top rated.  Cancellation_policy analysis:<jupyter_code>s_cp = get_percents(df_seattle_top, df_seattle_values, 'cancellation_policy')
s_cp.head(2)
s_cp.tail(2)
plot(s_cp)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10433719070467588, 'fraction_numerical': 0.009972716153918525, 'mean_word_length': 6.625538020086084, 'pattern_counts': {'":': 0, '<': 77, '<?xml version=': 0, '>': 79, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '590750', 'license': 'CC0-1.0', 'n_tokens_mistral': 3600, 'n_tokens_neox': 3474, 'n_words': 1009, 'path': '4 - Top rated analysis.ipynb', 'repo_name': 'gustavo-leandro/Airbnb-analysis-Boston-and-Seattle', 'size': '10629'}