<jupyter_start><jupyter_text>The CRISP-DM method will be applied here to keep track of the analysis process<jupyter_code># import libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
plt.style.use('seaborn')
import seaborn as sns
import geopandas<jupyter_output><empty_output><jupyter_text>1.Business UnderstandingI downloaded Berlin Airbnb dataset via http://insideairbnb.com/get-the-data.html.  For this project, I have been interested in using Berlin Airbnb dataset.I will focus on these questions below:!    1)Which place is the average most expensive accommodation in Berlin?2)What is the most expensive price on a room type basis?3)What is the distribution of prices on the basis of latitude and longitude?4)What is the most correlated three variable with price?5)Please you can create a model about make a prediction of room price.   Berlin City Mapresorce:http://ontheworldmap.com/germany/city/berlin/berlin-districts-map.html  2.Data Understanding Berlin city Maps is located on below side.<jupyter_code>#Downloaded https://gist.github.com/pfloh/ae03cdabca0c822d5283 geojson and convert name to Berlin_map
countries_gdf = geopandas.read_file("Berlin_map.geojson")
countries_gdf.plot()
# load dataset
listings_df = pd.read_csv('listings.csv')
listings_df.head()
# number of data
listings_df.shape[0]
# data format for each column
listings_df.info()
# What kind information is provied?
listings_df.columns
#describe
listings_df.describe()<jupyter_output><empty_output><jupyter_text>3. Prepare Data There are some necessary stpes to apply before continue exploring the dataset:Drop unused columnsConvert string values to numberHandle missing values, drop them if necessary<jupyter_code># Drop unused columns
columns_to_drop = ['host_name','last_review']

listings_df.drop(columns_to_drop, axis=1, inplace=True)
# Convert string to number
def listings_df_room_type(listings_df):
    """
    
    Convert room_type from words to integer for calculating the mean
    
    Parameters:
    listings_df: a dataframe that will be converted
    
    Returns:
    dataframe: a converted dataframe with room_type column becomes measurable
    
    """
    room_type_map = {
        'Entire home/apt' : 1,
        'Hotel room' : 2,
        'Private room' : 3,
        'Shared room' : 4,
        np.nan: np.nan
    }
    listings_df['room_type_num'] = listings_df['room_type'].apply(lambda x: np.nan if x == np.nan else room_type_map[x] )
    
    return listings_df
listings_df_room_type(listings_df)
listings_df.head()
# Convert string to number
def listings_df_neighbourhood_group(listings_df):
    """
    
    Convert neighbourhood_group from words to integer for calculating the mean
    
    Parameters:
    listings_df: a dataframe that will be converted
    
    Returns:
    dataframe: a converted dataframe with neighbourhood_group column becomes measurable
    
    """
    neighbourhood_group_map = {
'Friedrichshain-Kreuzberg' :1
,'Mitte' :2
,'Pankow' :3
,'Neukölln' :4
,'Charlottenburg-Wilm.' :5
,'Tempelhof - Schöneberg' :6
,'Lichtenberg' :7
,'Treptow - Köpenick' :8
,'Steglitz - Zehlendorf' :9
,'Reinickendorf' :10
,'Spandau' :11
,'Marzahn - Hellersdorf':12,
        np.nan: np.nan
    }
    listings_df['neighbourhood_group_num'] = listings_df['neighbourhood_group'].apply(lambda x: np.nan if x == np.nan else neighbourhood_group_map[x] )
    
    return listings_df
listings_df_neighbourhood_group(listings_df)
listings_df.head()
# missing values
listings_df.isnull().sum()
# drop columns
missing_columns_to_drop = ['name','reviews_per_month']

listings_df.drop(missing_columns_to_drop, axis=1, inplace=True)
# missing values check
listings_df.isnull().sum()<jupyter_output><empty_output><jupyter_text>4. Answer Questions base on dataset  Question 1Which place is the average most expensive accommodation in Berlin?<jupyter_code>listings_df.neighbourhood_group.value_counts()
red_square = dict(markerfacecolor='salmon', markeredgecolor='salmon', marker='.')

listings_df.boxplot(column='price', by='neighbourhood_group', 
           flierprops=red_square, vert=False, figsize=(10,8))

plt.xlabel('\nMedian Price', fontsize=12)
plt.ylabel('District\n', fontsize=12)
plt.title('\nBoxplot: Prices by Neighbourhood\n', fontsize=14, fontweight='bold')

# get rid of automatic boxplot title
plt.suptitle('');
def display_bar_chart (df,column, title):
    '''
    Displays a bar chart with a title
    
    Parameters:
    df: a dataframe
    column: the column which we want to show
    title: the title of the chart
    
    Returns:
    None
    
    '''
    status_vals=df[column].value_counts()
    (status_vals[:20]/df.shape[0]).plot(kind="bar");
    plt.title(title);
    
#Provide a pandas series of the counts for each Price
display_bar_chart(listings_df, "price", "What are the most twenty demanded room price in Berlin Airbnb ?")
#Provide a pandas series of the counts for each neighbourhood_group
display_bar_chart(listings_df, "neighbourhood_group", "Where are the most demanded rooms place in Berlin Airbnb ?")
#Compare selected neighbourhood_group based on average room price
Compare=listings_df.groupby(['neighbourhood_group'])['price'].mean()
Compare.plot(kind="bar")
Compare<jupyter_output><empty_output><jupyter_text>Charlottenburg-Wilm. is the average most expensive accomodation place in Berlin  Question 2What is the most expensive price on a room type basis?<jupyter_code>listings_df.room_type.value_counts()
#Compare selected room type based on average price
compare_room_type=listings_df.groupby(['room_type'])['price'].max()
compare_room_type.plot(kind='bar')
compare_room_type<jupyter_output><empty_output><jupyter_text>Entire home/apt is most expensive price on a room type basis. This price is 9.000€.   Question 3What is the distribution of prices on the basis of latitude and longitude?<jupyter_code>plot=listings_df.plot(kind="scatter", x="longitude", y="latitude", alpha=0.6, figsize=(8,6), 
        c="price", cmap="plasma", colorbar=True ,sharex=False);
plot<jupyter_output><empty_output><jupyter_text>Distribution of prices is on the basis of latitude and longitude on above side.  Question 4What is the most correlated three variable with price?<jupyter_code>ax = plt.subplots( figsize=(10,10) )
sns.heatmap(listings_df.corr(),annot=True,linewidths=1)
plt.show()<jupyter_output><empty_output><jupyter_text>Availability_365 is that the most correlated with price. Correlation is 0.12.   Question 5Please you can create a model about make a prediction of room price.<jupyter_code>from sklearn import datasets, linear_model
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
listings_df.columns
X=listings_df.drop(['id', 'host_id', 'neighbourhood_group', 'neighbourhood','room_type','price'],axis=1)
y=listings_df['price']
# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 42)
print('X_train Shape:', X_train.shape)
print('y_train Shape:', y_train.shape)
print('X_test Shape:', X_test.shape)
print('y_test Shape:', y_test.shape)
# Import the model we are using
from sklearn.ensemble import RandomForestRegressor
estimators = [2, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80,100]
# Instantiate model with 2-100 decision trees
for i in estimators:
    rf = RandomForestRegressor(n_estimators = i, random_state = 42)
# Train the model on training data
    rf.fit(X_train, y_train);
    print('estimators:',i)
# Use the forest's predict method on the test data
    predictions = rf.predict(X_test)
# Calculate the absolute errors
    errors = abs(predictions - y_test)
# Print out the mean absolute error (mae)
    print('Mean Absolute Error:', round(np.mean(errors), 2), 'degrees.')<jupyter_output>estimators: 2
Mean Absolute Error: 35.92 degrees.
estimators: 5
Mean Absolute Error: 34.55 degrees.
estimators: 10
Mean Absolute Error: 32.78 degrees.
estimators: 15
Mean Absolute Error: 31.85 degrees.
estimators: 20
Mean Absolute Error: 31.2 degrees.
estimators: 25
Mean Absolute Error: 30.98 degrees.
estimators: 30
Mean Absolute Error: 31.17 degrees.
estimators: 35
Mean Absolute Error: 30.86 degrees.
estimators: 40
Mean Absolute Error: 30.7 degrees.
estimators: 45
Mean Absolute Error: 30.44 degrees.
estimators: 50
Mean Absolute Error: 30.24 degrees.
estimators: 55
Mean Absolute Error: 30.08 degrees.
estimators: 60
Mean Absolute Error: 30.22 degrees.
estimators: 65
Mean Absolute Error: 30.23 degrees.
estimators: 70
Mean Absolute Error: 30.11 degrees.
estimators: 75
Mean Absolute Error: 30.14 degrees.
estimators: 80
Mean Absolute Error: 30.13 degrees.
estimators: 100
Mean Absolute Error: 29.82 degrees.<jupyter_text>We can choose estimators:100 and create model again.<jupyter_code># Instantiate model with 100 decision trees
rf = RandomForestRegressor(n_estimators = 100, random_state = 42)
# Train the model on training data
rf.fit(X_train, y_train);
# Use the forest's predict method on the test data
predictions = rf.predict(X_test)
# Calculate the absolute errors
errors = abs(predictions - y_test)
# Print out the mean absolute error (mae)
print('Mean Absolute Error:', round(np.mean(errors), 2), 'degrees.')<jupyter_output>Mean Absolute Error: 29.82 degrees.
{'fraction_non_alphanumeric': 0.09648273636656986, 'fraction_numerical': 0.026352586855975045, 'mean_word_length': 5.2909336941813265, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '502955', 'license': 'FTL', 'n_tokens_mistral': 3256, 'n_tokens_neox': 2964, 'n_words': 915, 'path': 'Berlin_Airbnb.ipynb', 'repo_name': 'erdutku/Write-A-Data-Science-Blog-Post', 'size': '9297'}