<jupyter_start><jupyter_text>Riley's Analysis Notebook March 12th 2021--- EDA<jupyter_code>import pandas as pd
import seaborn as sns
import matplotlib.pylab as plt
from functions import *
df = process_data("./../../data/raw/listings.csv")
df
sns.pairplot(df)
# What is the relationship between rating and price like? What does it look like when you remove outliers and focus on the bulk of observations?
sns.displot(data=df[df['property_type'] == "Entire house"], x="price", y="review_scores_rating")
df_trimmed = df[(df['price'] <= 2000) & (df['review_scores_rating'] >= 50)]
sns.displot(data=df_trimmed[df_trimmed['property_type'] == "Entire house"], x="price", y="review_scores_rating")
# I am interested in comparing the different neighbourhoods, so i would like to see what they are
sns.countplot(x="neighbourhood_cleansed", data=df)
plt.xticks(rotation=45)<jupyter_output><empty_output><jupyter_text>Research Question 1: Which neighbouhood in San Francisco should I choose for a high quality rental experience at a low price?<jupyter_code># First, because I know that I want a high rating and low price, I will plot the counts again with the lowest ratings and highest prices filtered out.
# EDIT: After doing this filtering I realized that it significantly changed the most popular neighbourhoods, so I decided not to do it,
# in order to preserve the validity of the data.
# I will leave the dataset as is, and just take the top 10 most common neighbourhoods.
sns.countplot(x="neighbourhood_cleansed", data=df, order=df['neighbourhood_cleansed'].value_counts().iloc[:10].index)
plt.xticks(rotation=45)
# Based on these results, I will filter my dataset to only include these 10 neighbourhoods
df_reduced = popular_locations(df)
# I will use a coutplot again to confirm that the new data looks okay.
sns.countplot(x="neighbourhood_cleansed", data=df_reduced)
plt.xticks(rotation=45)
# Here, I will remove the outliers to give a more focused look at the bulk of the data, but I would like to note 
# that there are some listings which cause the data in most of the neighbourhoods to spike down around 20

sns.violinplot(x="neighbourhood_cleansed", y="review_scores_rating", data=df_reduced[df_reduced['review_scores_rating'] >= 60])
plt.xticks(rotation=45)
# I would also like to take a look at the relationship between price and neighbourhood. 
# Here, I will remove the outliers to give a more focused look at the bulk of the data, but I would like to note 
# that there are some listings which cause the data in a few neighbourhoods to spike up near $10,000

sns.violinplot(x="neighbourhood_cleansed", y="price", data=df_reduced[df_reduced['price'] <= 2500])
plt.xticks(rotation=45)<jupyter_output><empty_output><jupyter_text>From these 2 plots comparing the relationships between rating & neighbourhood, and price & neighbourhood, As well as what I saw in my EDA, it looks like the average rental experience on Airbnb is relativly affordable, and meets the expactations of the customer. I the violin plots, you can see that many of the plots spike downwards for rating, and upwards for price, indicating that there are some negative experiences, and some expensive listings, but the majority of the data tends to be quite similar from neighbourhood to neighbourhood.There is something that I feel is worth noting from these charts; The short and fat shapes of "Castro / Upper Market" in rating vs neighbourhood, and of "Bernal Heights" and "Outer Sunset" in price vs neighbourhood. This indicated that these variables are more consistent across listings in these neighbourhoods, because the data tends to clump towards similar values. This is good from our perspective as customers because there is a higher probability that we will get what we are looking for.<jupyter_code># Based on these violin charts, I will look more closely at the 3 neighbourhoods which stand out to me 
# as having consistently good ratings and low prices. Now that I have identified the neighborhoods which appear
# to be most consistent, I will compare the neighborhoods more closely to try to find the best balance betweeen 
# price and rating.

df_reduced = chosen_locations(df)

sns.kdeplot(
    data=df_reduced,
    x="price",
    y="review_scores_rating",
    hue="neighbourhood_cleansed",
    thresh=.1
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0711645101663586, 'fraction_numerical': 0.010397412199630314, 'mean_word_length': 5.5, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '233726', 'license': 'MIT', 'n_tokens_mistral': 1279, 'n_tokens_neox': 1176, 'n_words': 561, 'path': 'analysis/riley/.ipynb_checkpoints/analysis-checkpoint.ipynb', 'repo_name': 'data301-2020-winter2/course-project-group_1048', 'size': '4328'}