<jupyter_start><jupyter_text>Exploring Detroit Demolitions of Vacant Properties Accompanied with Linear Regression and Random Forest--- An analysis was performed on Detroit's demolition dataset. Demolitions done by the Detroit Land Bank Authority or Detroit Building Authority since January 1, 2014. Projects listed in this dataset have a known price, demolition knock-down date, and demolition contractor.   This analysis utilizes data science techniques, including:*   Exploratory Data Analysis (EDA)*   Visualization*   Correlation Analysis*   Linear Regression*   Random Forest  Data source:https://data.detroitmi.gov/Property-Parcels/Detroit-Demolitions/rv44-e9di  Loading the Data<jupyter_code>import pandas as pd
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns

# Data transformed to Github raw data for reporduction purposes. 
# Read the data into a Pandas DataFrame.
detDemos = pd.read_csv('https://raw.githubusercontent.com/WorldCapital/Detroit-Demolitions/master/Detroit_Demolitions.csv', parse_dates=[4])<jupyter_output><empty_output><jupyter_text>Exploring the Data<jupyter_code># Get some summaries of columns and data types.
print(detDemos.info())<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 18764 entries, 0 to 18763
Data columns (total 16 columns):
Address                   18764 non-null object
Parcel ID                 18764 non-null object
Contractor Name           18764 non-null object
Price                     18764 non-null float64
Primary Funding Source    18764 non-null object
Demolition Date           18764 non-null object
Commercial Building       18764 non-null object
Council_District          18764 non-null int64
Neighborhood              18764 non-null object
Latitude                  18764 non-null float64
Longitude                 18764 non-null float64
Location                  18764 non-null object
Street Direction          845 non-null object
Street Number             18764 non-null int64
Street Name               18764 non-null object
Street Type               18513 non-null object
dtypes: float64(3), int64(2), object(11)
memory usage: 2.3+ MB
None<jupyter_text>18,764 dated demolitions in the city of Detroit.<jupyter_code>print(detDemos.head())
detDemos.isna().sum()
detDemos.fillna(0, inplace=True)
detDemos.describe()<jupyter_output><empty_output><jupyter_text>The average price for all demolitions is 17,062.  Contracting companies<jupyter_code>print(len(detDemos['Contractor Name'].unique()))<jupyter_output>35<jupyter_text>There are 35 different contractors working demolitions in Detroit.<jupyter_code>contractors = detDemos.groupby(['Contractor Name'], as_index=True)

# size gives the number of contracts per company
print(contractors.size())<jupyter_output>Contractor Name
1 Way Service                             83
313 Construction, LLC                     40
ABC Demolition                           445
Able Demolition                         1995
Adamo Group, Inc.                       3355
BBEK Environmental                         1
Berkshire Development, Inc.               13
Blackstar                                  1
Blue Star                                716
Brown & Glo Wrecking                      23
Brown Environmental Construction         266
DMC Consultants                         2062
Den-Man Contractors                      582
Direct Construction Services              93
Dore & Associates                        189
Esso Wrecking Co.                        217
Farrow Group                             243
Futurenet Group                           88
GLO WRECKING CO                           18
Gayanga Co                               347
Homrich                                 3311
J Keith Construction                   [...]<jupyter_text>Top contractors for demolitions<jupyter_code>detDemos['Contractor Name'].value_counts().head(10)
print(detDemos['Price'].mean())<jupyter_output>17062.434321040317<jupyter_text>Neighborhoods with the highest demolitions<jupyter_code>detDemos['Neighborhood'].value_counts().head(15)<jupyter_output><empty_output><jupyter_text>Top contractors with the largest prices<jupyter_code>n=5
contractors_sum = contractors.sum();
top_n_contractors_df = contractors_sum.nlargest(n,'Price')
print(top_n_contractors_df)

top_n_contractor_names = top_n_contractors_df.index.values

top_n_contractors = detDemos[detDemos['Contractor Name'].isin(top_n_contractor_names)]
top_n_contractors.groupby(['Contractor Name']).size()<jupyter_output><empty_output><jupyter_text>Mean prices of top contractors<jupyter_code>top_n_contractors.groupby(['Contractor Name'])['Price'].mean()
print(top_n_contractors['Price'].mean())<jupyter_output>16765.83791660007<jupyter_text>Average price for top contractors is $16,765.  Visualization<jupyter_code>detDemos['Neighborhood'].value_counts().head(25).plot(kind = 'bar', figsize=(10,6), color=('black'))
plt.title('Total Number (#) of Vacant Properties for Demolition by Neighborhood')

detDemos['Contractor Name'].value_counts().head(25).plot(kind = 'barh', figsize=(10,6), color=('blue'))
plt.title('Total Number (#) of Demolitions by Contractor')
plt.gca().invert_yaxis()


from pylab import rcParams
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn

rcParams['figure.figsize'] = 12, 8
seaborn.set(rc={'axes.facecolor':'lightgrey', 'figure.facecolor':'white'})

a = detDemos['Street Number'] 

_ = plt.hist(a, bins='auto')  # arguments are passed to np.histogram
plt.title("Histogram of Detroit Demolitions by Street Number")
plt.show()






ax = sns.regplot(x=contractors.size(),fit_reg=False, y=contractors_sum['Price'], color="g", scatter_kws={"s":contractors_sum['Price']/150000})
# we reduce the size of the dot for the scatter plot by a factor of 100000 for good visibility
plt.ticklabel_format(style = 'plain')  # remove exponential labels with this setting
ax.set(xlabel='Total Contracts', ylabel='Total Price')
from pylab import rcParams
rcParams['figure.figsize'] = 12, 8
g = sns.boxplot(detDemos['Price'])
print("median price ", detDemos['Price'].median())
print("mean price ", detDemos['Price'].mean())
plt.title('Box Plot of Detroit Demolition Street Numbers')
plt.ylabel('Street Numbers (#)')
detDemos["Street Number"].plot.box();
plt.title('Box Plot of Latitude Coordinates')
plt.ylabel('Latitude')
detDemos["Latitude"].plot.box();
plt.title('Box Plot of Longitude Coordinates')
plt.ylabel('Longitude')
detDemos["Longitude"].plot.box();
sns.pairplot(detDemos,kind="reg")
#plt.title('Correlation Pairplot of Michigan Income and Obesity')
plt.show()
# Now to plot the geo distribution of the top n contractors demolition sites
top_n_contractors = detDemos[detDemos['Contractor Name'].isin(top_n_contractor_names)]
sns.lmplot(x='Longitude', y='Latitude', data=top_n_contractors,col="Contractor Name",
           fit_reg=False, # No regression line
           scatter_kws={"s": 10})   # Color for dots

plt.show()<jupyter_output><empty_output><jupyter_text>Comparison of resident vs commerical demolitions by top contractor<jupyter_code>sns.factorplot(x="Commercial Building", y="Price",
                    col='Contractor Name',
                    ci=None,
                    data=detDemos[detDemos['Contractor Name'].isin(top_n_contractor_names)], kind="bar",
                    size=4, aspect=.9);
unstacked_df = top_n_contractors.groupby(['Contractor Name', 'Commercial Building'])['Contractor Name'].count().unstack('Commercial Building')
print(unstacked_df)
unstacked_df.plot(kind='bar', stacked=True)<jupyter_output>Commercial Building         No  Yes
Contractor Name                    
Able Demolition           1889  106
Adamo Group, Inc.         3287   68
DMC Consultants           2016   46
Homrich                   3265   46
Rickman Enterprise Group  1752   43<jupyter_text>Demolitions of top contractors indicate that the significant majority of demolitions occur on resident properties as opposed to commerical.   Correlation Analysis<jupyter_code>detDemos.corr()
detDemos.corr()['Price'].sort_values(ascending=False)
detDemos.corr()['Street Number'].sort_values(ascending=False)
# Set up the matplotlib figure
f, ax = plt.subplots(figsize=(12, 10))
plt.title('Pearson Correlation of Detroit Demolitions')

# Draw the heatmap using seaborn
sns.heatmap(detDemos.corr(),linewidths=0.25,vmax=1.0, square=True, cmap="YlGnBu", linecolor='black', annot=True)<jupyter_output><empty_output><jupyter_text>Linear Regression<jupyter_code>numerical_columns = detDemos.dtypes[detDemos.dtypes != 'object'].index
numerical_data = detDemos[numerical_columns]

# we drop aspect_ratio, as it doesn't provide any useful info
#numerical_data.drop('Neighborhood', axis=1, inplace=True)
numerical_data.head(2)
from sklearn import tree
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.model_selection import train_test_split

train, test = train_test_split(numerical_data, test_size=0.2)
target_train = train.pop('Street Number')
target_test = test.pop('Street Number')
print('Train data: {} / {} = {}'.format(len(train), len(numerical_data), float(len(train)/len(numerical_data))))
print('Test data: {} / {} = {}'.format(len(test), len(numerical_data), float(len(test)/len(numerical_data))))
model = LinearRegression()
model.fit(train, target_train)
prediction = model.predict(test)
import matplotlib.lines as mlines
import matplotlib.transforms as mtransforms

# The mean squared error
print("Mean squared error: %.2f" % mean_squared_error(target_test, prediction))

# Explained variance score: 1 is perfect prediction
print('Accuracy: %.2f' % r2_score(target_test, prediction))

# Plot outputs
plt.scatter(target_test, prediction,  color='black')

line = mlines.Line2D([70, 71], [70, 71], color='red')
transform = ax.transAxes
line.set_transform(transform)
ax.add_line(line)


plt.show()<jupyter_output>Mean squared error: 13705298.87
Accuracy: 0.62<jupyter_text>Linear regression   Random Forest<jupyter_code>forest = RandomForestRegressor(
    max_depth=25,
    min_samples_split=15,
    n_estimators=1000,
    random_state=1)

forest.fit(train, target_train)
forest.feature_importances_
forest_prediction = forest.predict(test)
# The mean squared error
print("Mean squared error: %.2f" % mean_squared_error(target_test, forest_prediction))

# Explained variance score: 1 is perfect prediction
print('Accuracy: %.2f' % r2_score(target_test, forest_prediction))

# Plot outputs
plt.scatter(target_test, forest_prediction,  color='black')
# plt.plot(test, prediction, color='blue', linewidth=3)


plt.show()<jupyter_output>Mean squared error: 2302625.75
Accuracy: 0.94
{'fraction_non_alphanumeric': 0.10203320276067898, 'fraction_numerical': 0.03422868867748555, 'mean_word_length': 4.218004866180048, 'pattern_counts': {'":': 2, '<': 50, '<?xml version=': 0, '>': 50, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '303342', 'license': 'MIT', 'n_tokens_mistral': 3596, 'n_tokens_neox': 3209, 'n_words': 808, 'path': 'Detroit_Demolitions.ipynb', 'repo_name': 'kyle-w-brown/Detroit-Demolitions', 'size': '10722'}