<jupyter_start><jupyter_text>COVID-19 and Real Estate IntroductionThe COVID-19 significantly impacted the U.S. economy, with millions of Americans already out of work because of the crisis. In 1 year, women globally lost $800 billion in income due to Covid-19. Nearly 90 percent of New York City bar and restaurant owners couldn’t pay their rent in August. Health concerns and stay-at-home orders led to fewer buyers looking for homes and fewer sellers willing to list their properties or allow strangers to enter their homes during a pandemic. It looks like this year could be a buyer’s market for real estate. Will property prices fall due to Coronavirus impact? Should you buy a home right now? What causes real estate prices to increase?In this project we will try to give some answers to the previous questions. We will use a Dataset available in Kaggle, which contains Realtor.com real estate monthly housing Data from 2016 to 2021, broken by State and zip code, including median list price, average list price, median days on market, total active listings, new listings, price increases, price reductions, median listing price per square foot, etc. We The CRISP-DM Process (Cross Industry Process for Data Mining)The lessons leading up to the first project are about helping you go through CRISP-DM in practice from start to finish. Even when we get into the weeds of coding, try to take a step back and realize what part of the process you are in, and assure that you remember the question you are trying answer and what a solution to that question looks like.1. Business Understanding2. Data Understanding3. Prepare Data4. Data Modeling5. Evaluate the Results6. Deploy Business Undestanding1. Have house prices fallen in the US?2. Which field has more effect in increasing the price of the house?3. How the COVID-19 is affecting the housing Market? Data Understanding<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score, mean_squared_error
import seaborn as sns
%matplotlib inline

##We start gathering information from the dataset RDC_Inventory_Core_Metrics_Zip_History.csv
df = pd.read_csv('./RDC_Inventory_Core_Metrics_Zip_History.csv')
df

#Getting a concise summary of the DataFrame
df.info()
#We also see some stats numbers to undertand the numerical features
df.describe()<jupyter_output><empty_output><jupyter_text>Prepare Data<jupyter_code>#we need drop 913083 row with comments from authors
df = df.drop(913083)
#we have to drop unnecessary columns
df2 = df[['postal_code','zip_name','month_date_yyyymm','active_listing_count','average_listing_price',
          'total_listing_count','pending_ratio']]
df2['month_date_yyyymm_int'] = df2['month_date_yyyymm'].astype(np.int64)
df2
#this is datefrme for modeling, wi will not aggregte data
df3 = df[['postal_code','month_date_yyyymm','active_listing_count','average_listing_price',
          'total_listing_count','pending_ratio']]
#we need co convert columns for modeling
df3['month_date_yyyymm_int'] = df3['month_date_yyyymm'].astype(np.int64)
df3['postal_code_int'] = df3['postal_code'].astype(np.int64)
df3
#I supose we should to add column covid-time, to mark which dates are in this period
#df3['covid_time'] = np.where(df3['month_date_yyyymm'].astype(np.int64) >= 201903, 1, 0)
#sometimes it's better to use functions
def is_covid(date_int):
    ''' return int 1 or 0 
    Parameters:
        date_int - date format yyyymm with type integer'''
    
    if date_int>=201903:
        return 1
    else:
        return 0

#lets check docstrings
print (is_covid.__doc__)
df3['covid_time'] = df3.apply(lambda x: is_covid(x['month_date_yyyymm_int']),axis=1)
#I supose we should to add column covid-time, to mark which dates are in this period
#df2['covid_time'] = np.where(df2['month_date_yyyymm'].astype(np.int64) >= 201903, 1, 0)
df2['covid_time'] = df2.apply(lambda x: is_covid(x['month_date_yyyymm_int']),axis=1)
df2
#Drop the rows where at least one element is missing. 
df2 = df2.dropna()
df3 = df3.dropna()

#and an aggregation 
df_zip = df2.groupby(['zip_name', 'month_date_yyyymm'],as_index=False, sort=False).mean()
#For data scientists, checking correlations is an important part of the exploratory data analysis process, so in this part
#we plot a heatmap in order to check correlations between numerical features
sns.set(rc={'figure.figsize':(15,10)})
sns.heatmap(df_zip.corr(), annot=True, fmt=".2f",);<jupyter_output><empty_output><jupyter_text>Data Modeling<jupyter_code>X = df3.drop('average_listing_price', axis=1)
y = df3['average_listing_price']

#Split into train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

lm_model = LinearRegression(normalize=True) # Instantiate
lm_model.fit(X_train, y_train) #Fit

#Predict using your model
y_test_preds = lm_model.predict(X_test)
y_train_preds = lm_model.predict(X_train)

#Score using your model
test_score = r2_score(y_test, y_test_preds)
train_score = r2_score(y_train, y_train_preds)
train_score<jupyter_output><empty_output><jupyter_text>Evaluate the Results Question1 Have house prices fallen in the US?<jupyter_code>#we have to look on the plot with average_listing_price filed
plt.plot('month_date_yyyymm', 'average_listing_price', data=df_zip.loc[df_zip.zip_name=='florida, ny', :],label='florida, ny', color='tab:green')
plt.plot('month_date_yyyymm', 'average_listing_price', data=df_zip.loc[df_zip.zip_name=='miami, fl', :], label='miami, fl', color='tab:red')

plt.legend()
plt.show()
plt.plot('month_date_yyyymm', 'average_listing_price', data=df_zip.loc[df_zip.zip_name=='texas city, tx', :],label='texas city, tx',color='tab:grey')
plt.plot('month_date_yyyymm', 'average_listing_price', data=df_zip.loc[df_zip.zip_name=='california city, ca', :],label='california city, ca',color='tab:orange')
plt.legend()
plt.show()
def calc_interest(zip_name, field):
    ''' print calculated percents 
    Parameters:
        zip_name - string, zip name
        field - string, name of filed which need to calc'''
    #divide datasets on 2: pre-covid and covid
    data_pre_covid = df_zip.loc[(df_zip.zip_name==zip_name) & (df_zip.covid_time==0), :].sort_values(by=['month_date_yyyymm_int'])
    data_covid = df_zip.loc[(df_zip.zip_name==zip_name) & (df_zip.covid_time==1), :].sort_values(by=['month_date_yyyymm_int'])
    #find prices of start and end of periods
    price_pre_start = data_pre_covid.iloc[0][field]
    price_pre_end = data_pre_covid.iloc[-1][field]
    price_post_start = data_covid.iloc[0][field]
    price_post_end = data_covid.iloc[-1][field]
    p_pre = 100 - price_pre_start/price_pre_end*100
    p_post = 100 - price_post_start/price_post_end*100
    print('How have ', field,' changed in ',zip_name,' in pre-covid time: ', p_pre)
    print('How have ', field,' changed in ',zip_name,' in covid-time: ', p_post)
    
#lets check numbers
calc_interest('florida, ny','average_listing_price')
calc_interest('miami, fl','average_listing_price')
calc_interest('texas city, tx','average_listing_price')
calc_interest('california city, ca','average_listing_price')<jupyter_output>How have  average_listing_price  changed in  florida, ny  in pre-covid time:  -2.3972442002850443
How have  average_listing_price  changed in  florida, ny  in covid-time:  36.501513952106045
How have  average_listing_price  changed in  miami, fl  in pre-covid time:  5.98363587113586
How have  average_listing_price  changed in  miami, fl  in covid-time:  24.620014686662188
How have  average_listing_price  changed in  texas city, tx  in pre-covid time:  2.849166833549063
How have  average_listing_price  changed in  texas city, tx  in covid-time:  15.810859288630468
How have  average_listing_price  changed in  california city, ca  in pre-covid time:  18.038292368306614
How have  average_listing_price  changed in  california city, ca  in covid-time:  35.388903799128684<jupyter_text>Answer1: Not at all. In the plot above, we can observe an increasing trend of average prices over the years.House prices are rising in general in the most states, but we can see clearly the difference between 2 periods: pre-covid time and covid-time. For example, average prices have been decreasing since 201607 - 201903 in 'florida, ny' on ~2.4%, and they significantly have been increasing by 36.50 since 201903 - 202107, after that covid happened. Question2: Which field has more effect in increasing the price of the house?<jupyter_code>df_zip.corr()['average_listing_price']<jupyter_output><empty_output><jupyter_text>Answer2: After finding the correlation of the average price with all other fields we got that active_listing_count has the highest effect in increasing the price of the houses.This information will help us to understand how covid-19 impacts house prices.  Question3: How the COVID-19 is affecting the housing Market?<jupyter_code>plt.plot('month_date_yyyymm', 'active_listing_count', data=df_zip.loc[df_zip.zip_name=='florida, ny', :],label='florida, ny', color='tab:green')
plt.plot('month_date_yyyymm', 'active_listing_count', data=df_zip.loc[df_zip.zip_name=='miami, fl', :], label='miami, fl', color='tab:red')
plt.plot('month_date_yyyymm', 'active_listing_count', data=df_zip.loc[df_zip.zip_name=='texas city, tx', :],label='texas city, tx',color='tab:grey')
plt.plot('month_date_yyyymm', 'active_listing_count', data=df_zip.loc[df_zip.zip_name=='california city, ca', :],label='california city, ca',color='tab:orange')

plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>Answer3: This graph shows that active_listing_count is reducing in most states.<jupyter_code>#lets check numbers
calc_interest('florida, ny','active_listing_count')
calc_interest('miami, fl','active_listing_count')
calc_interest('texas city, tx','active_listing_count')
calc_interest('california city, ca','active_listing_count')<jupyter_output>How have  active_listing_count  changed in  florida, ny  in pre-covid time:  -73.33333333333334
How have  active_listing_count  changed in  florida, ny  in covid-time:  57.57575757575758
How have  active_listing_count  changed in  miami, fl  in pre-covid time:  13.979976953409931
How have  active_listing_count  changed in  miami, fl  in covid-time:  -111.52654786633138
How have  active_listing_count  changed in  texas city, tx  in pre-covid time:  1.470588235294116
How have  active_listing_count  changed in  texas city, tx  in covid-time:  -59.756097560975604
How have  active_listing_count  changed in  california city, ca  in pre-covid time:  52.0
How have  active_listing_count  changed in  california city, ca  in covid-time:  -201.72413793103448<jupyter_text>Active home listings hit record low. It explains why home prices are through the roof.For example, active listings have been increasing since 2016/07–2019/03 in 'miami, fl' on ~13.98%, and they significantly have been decreasing on 111.53% since 2019/03–2021/07. Evaluation of prediction<jupyter_code>### our predictions is not good, because we dont have enought information in the dataset

preds_vs_act = pd.DataFrame(np.hstack([y_test.values.reshape(y_test.size,1), y_test_preds.reshape(y_test.size,1)]))
preds_vs_act.columns = ['actual', 'preds']
preds_vs_act['diff'] = preds_vs_act['actual'] - preds_vs_act['preds']
preds_vs_act


plt.plot(preds_vs_act['preds'], preds_vs_act['diff'], 'bo');
plt.xlabel('predicted');
plt.ylabel('difference');<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09756731018959397, 'fraction_numerical': 0.04042940005194356, 'mean_word_length': 5.963230861965039, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '643935', 'license': 'MIT', 'n_tokens_mistral': 4178, 'n_tokens_neox': 3781, 'n_words': 1164, 'path': 'Project1.ipynb', 'repo_name': 'kirtoka/real_estate_covid', 'size': '11551'}