<jupyter_start><jupyter_text>Load data<jupyter_code>df = pd.read_csv('../../Data/raw/listings.csv')
df
df.describe()<jupyter_output><empty_output><jupyter_text>Drop unecessary columns<jupyter_code>df_drop=df.drop(['neighbourhood_group','last_review'],axis=1)  #removed the empty neighbourhood_group column

df_drop.describe()
#df.dtypes
#for cols in list(df_drop):
#    if df_drop[cols].dtypes == 'object':
#        df_drop[cols].str.lower()
#df_drop = df_drop.apply(lambda x: x.str.lower() if(x.dtype()==str) else x) 
#df_drop['name'].str.lower()
#df_drop['name']
#df1 = df_drop.apply(lambda x: x.str.lower() if x.dtype=='object' else x)
df_drop
df1 = df_drop.applymap(lambda x:x.lower() if type(x) == str else x)
df1<jupyter_output><empty_output><jupyter_text>Finding Outliers<jupyter_code>df_drop.describe()
#boxplots will help us determine outliers.

sns.boxplot(x=df_drop['price'])
sns.boxplot(x=df_drop['minimum_nights'])
sns.scatterplot(x='price',y='minimum_nights',data = df_drop)
df_drop.loc[df_drop['minimum_nights']>400,'minimum_nights'] = df_drop['minimum_nights'].mean()
df_drop.loc[df_drop['price']>150000,'price'] = df_drop['price'].mean()

sns.scatterplot(x='price',y='minimum_nights',data = df_drop)
df_drop.describe().T
def load_and_process(csv_file_path):

    # Method Chain 1 (Load data and deal with missing data, drop unnecessary columns,and convert all strings to lowercase)

    df1 = (
          pd.read_csv(csv_file_path)
          .dropna(axis=1, how='all')                        #drops the neighbourhood column where all whole column is empty.
          .drop(['last_review','reviews_per_month'],axis=1)
          .applymap(lambda x:x.lower() if type(x) == str else x)
      )
    
    #Using boxplot and scatter plots, the outliers are determined.
    
    #sns.boxplot(x=df1['minimum_nights'])
    #sns.scatterplot(x='price',y='minimum_nights',data = df1)
    #sns.boxplot(x=df1['price'])
    
    df1.loc[df1['minimum_nights']>400, 'minimum_nights'] = df1['minimum_nights'].mean()
    df1.loc[df1['price']>150000,'price'] = df1['price'].mean()
    

    return df1<jupyter_output><empty_output><jupyter_text>EDA    **About:**The data in this dataset is regarding the airbnb listings in Dublin, Ireland. The original dataset contains 15 columns and 19277 rows. These 15 columns (i.e variables) are either categorical or numerical data or string objects. The dataset contains very useful information about the name of the Airbnbs, the name of the hosts who owns the Airbnb, the location of the Airbnb, the price per night, the minimum nights, availability, etc.  Dataset Cleaning and Loading<jupyter_code>df_bnb=load_and_process('../../Data/raw/listings.csv')
df_bnb<jupyter_output><empty_output><jupyter_text>Exploring the dataset<jupyter_code>df_bnb.shape
df_bnb.info()
df_bnb[df_bnb['host_name'].isnull()]
df_bnb.describe(include=np.number).T
df_bnb.describe(include='object').T<jupyter_output><empty_output><jupyter_text>Key Observations:- ward 115 is the most popular location for bnbs.- the most listed room type is an entire home or an apartment- the minimum nights of stay varies from 1 day to 365 days( or 1 year)- if the the availability is less, it means the bnb listing is more popular  Potential areas of interest:- What is relation between price and room type?- What is the relation between price and number of nights?- What is the relation between price and neighbourhood?- What is the relation between price and number of reviews?- What is the relation between price and availability? - What is the relation between the number of nights and the room type?<jupyter_code>#sns.heatmap(df_bnb.select_dtypes(exclude=['object']))
import matplotlib.pyplot as plt
fig,ax=plt.subplots(figsize=(10,10))
corr = df_bnb.corr()# plot the heatmap
sns.heatmap(corr, xticklabels=corr.columns, yticklabels=corr.columns, annot=True, cmap=sns.diverging_palette(600, 500, as_cmap=True), ax=ax)
sns.scatterplot(x='price',y='minimum_nights',data=df_bnb)

title = 'Room type location based on longitude and latitude'
plt.figure(figsize=(10,6))
sns.scatterplot(df_bnb.longitude,df_bnb.latitude,hue=df_bnb.room_type).set_title(title)
plt.ioff()
import warnings
warnings.filterwarnings(action='ignore')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1277046095954845, 'fraction_numerical': 0.015522107243650047, 'mean_word_length': 5.6976377952755906, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '78379', 'license': 'MIT', 'n_tokens_mistral': 1532, 'n_tokens_neox': 1446, 'n_words': 357, 'path': 'Analysis/Sanjana/.ipynb_checkpoints/milestone2-Copy1-checkpoint.ipynb', 'repo_name': 'data301-2020-winter2/course-project-group_1025', 'size': '4252'}