<jupyter_start><jupyter_text>Descriptive MethodThe descriptive method is being computed. This section might take 1 minute to display images.  Visualize Descriptive ClustersVisualize the clustering results of our data using scatter plots<jupyter_code>## Compute t-sne clusters for the descriptive method
# takes one minute for this cell to finish running
import matplotlib.pyplot as plt
from sklearn.manifold import TSNE

tsne = TSNE(n_components=2, verbose=1, perplexity=30, n_iter=1000, learning_rate=200)
tsne_scale_results = tsne.fit_transform(df.iloc[:, 3:10])

tsne_df_scale = pd.DataFrame(tsne_scale_results, columns=['tsne1', 'tsne2'])

plt.figure(figsize = (10,10))
plt.scatter(tsne_scale_results[:,0],tsne_scale_results[:,1],alpha=0.25, facecolor='lightslategray')
plt.xlabel('Sale Month')
plt.ylabel('Sale Price')
plt.legend()
plt.show()

## calculate k-means clusters from
from sklearn.cluster import KMeans
import seaborn as sns

# apply kmeans to t-sne clusters.
sse = []
k_list = range(1, 15)

for k in k_list:
    km = KMeans(n_clusters=k)
    km.fit(tsne_df_scale)
    sse.append([k, km.inertia_])
    
tsne_results_scale = pd.DataFrame({'Cluster': range(1,15), 'SSE': sse})

plt.figure(figsize = (10,10))

kmeans_tsne_scale = KMeans(n_clusters=4, n_init=100, max_iter=400, init='k-means++', random_state=42).fit(tsne_df_scale)
labels_tsne_scale = kmeans_tsne_scale.labels_

clusters_tsne_scale = pd.concat([tsne_df_scale, pd.DataFrame({'tsne_clusters':labels_tsne_scale})], axis=1)
sns.scatterplot(clusters_tsne_scale.iloc[:,0],clusters_tsne_scale.iloc[:,1],hue=labels_tsne_scale, palette='Set1', s=100, alpha=0.6).set_title('Cluster Vis tSNE Scaled Data', fontsize=15)
plt.legend()
plt.show()<jupyter_output>[t-SNE] Computing 91 nearest neighbors...
[t-SNE] Indexed 12683 samples in 0.024s...
[t-SNE] Computed neighbors for 12683 samples in 0.214s...
[t-SNE] Computed conditional probabilities for sample 1000 / 12683
[t-SNE] Computed conditional probabilities for sample 2000 / 12683
[t-SNE] Computed conditional probabilities for sample 3000 / 12683
[t-SNE] Computed conditional probabilities for sample 4000 / 12683
[t-SNE] Computed conditional probabilities for sample 5000 / 12683
[t-SNE] Computed conditional probabilities for sample 6000 / 12683
[t-SNE] Computed conditional probabilities for sample 7000 / 12683
[t-SNE] Computed conditional probabilities for sample 8000 / 12683
[t-SNE] Computed conditional probabilities for sample 9000 / 12683
[t-SNE] Computed conditional probabilities for sample 10000 / 12683
[t-SNE] Computed conditional probabilities for sample 11000 / 12683
[t-SNE] Computed conditional probabilities for sample 12000 / 12683
[t-SNE] Computed conditional probabilities for sam[...]<jupyter_text>Visualize a grid of the seperate properties of each of our 4 distinct k-means clusters using histograms<jupyter_code>import math
import matplotlib.ticker
import seaborn as sns

cluster_tsne_profile = pd.merge(df, clusters_tsne_scale['tsne_clusters'], left_index=True, right_index=True)

from functools import partial
bin_hist = partial(plt.hist,bins=20)
bin_hist.__name__ = "bin_hist"
bin_hist.__module__ = plt.hist.__module__

for c in cluster_tsne_profile:
    
    # sale month is for regression only, we use sale date instead
    if(c == "SALE MONTH" or c == "RESIDENTIAL UNITS"):
        continue

    grid = sns.FacetGrid(cluster_tsne_profile, col='tsne_clusters')
    grid.map(bin_hist, c)
    
    if(c == "SALE DATE" or c == "ZIP CODE" or c == "LAND SQUARE FEET" or c == "GROSS SQUARE FEET"):
        grid.set_xticklabels(rotation=-30)
        
    if(c == "BUILDING CLASS AT TIME OF SALE"):
        grid.set_xticklabels(fontsize=7)<jupyter_output><empty_output><jupyter_text>Visualize the multi-variable graph using scatter plots<jupyter_code>plt.figure(figsize=(5,10))
fig, (ax1,ax2) = plt.subplots(1, 2, figsize=(20,15))

# left graph
sns.scatterplot(data=cluster_tsne_profile, x='YEAR BUILT', y='GROSS SQUARE FEET', hue='tsne_clusters', s=85, alpha=0.4, palette='bright', ax=ax1).set_title('(tSNE) Clusters by Year Built and Size', fontsize=18)

# right graph.
sns.scatterplot(data=cluster_tsne_profile, x='GROSS SQUARE FEET', y='SALE PRICE', hue='tsne_clusters', s=85, alpha=0.4, palette='bright', ax=ax2).set_title('(tSNE) Clusters by Gross Square Feet and Price', fontsize=18)<jupyter_output><empty_output><jupyter_text>Predictive Method Method accuracy report:<jupyter_code>## Predictive method using deep forest regression
#train the deep forest model

X = df.iloc[:, 3:9].values
y = df.iloc[:, 9].values

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

from sklearn.ensemble import RandomForestRegressor

regressor = RandomForestRegressor(n_estimators=1000, random_state=0)
regressor.fit(X_train, y_train)
y_pred = regressor.predict(X_test)

#evaluate the model
from sklearn import metrics

print('R2:', metrics.r2_score(y_test, y_pred)) #coefficient of determination
print('Mean Absolute Error:', metrics.mean_absolute_error(y_test, y_pred))
print('Mean Squared Error:', metrics.mean_squared_error(y_test, y_pred))
print('Root Mean Squared Error:', np.sqrt(metrics.mean_squared_error(y_test, y_pred)))
print('Max Error:', metrics.max_error(y_test, y_pred))<jupyter_output>R2: 0.46003940688969813
Mean Absolute Error: 182013.87350262763
Mean Squared Error: 599892130725.1577
Root Mean Squared Error: 774527.03679417
Max Error: 26417341.821000002<jupyter_text>Predictive Method GUIUse a deep forest model to predict how much a single-family house would cost depending on the location, month of sale, and unit specifications.<jupyter_code># UI
import ipywidgets as widgets

wide_layout=widgets.Layout(width='40%', height='30px')
med_layout=widgets.Layout(width='32%', height='30px')
label_layout={'description_width': 'initial'}
## Month slider
user_month = widgets.SelectionSlider(
    options=[('January', 1),
             ('February', 2),
             ('March', 3),
             ('April', 4),
             ("May", 5),
             ("June", 6),
             ("July", 7),
             ("August", 8),
             ("September", 9),
             ("October", 10),
             ("November", 11),
             ("December",12)],
    value=9,
    description='Month bought in 2016/2017:',
    disabled=False,
    continuous_update=False,
    orientation='horizontal',
    readout=True,
    style=label_layout,
    layout=wide_layout
)

display(user_month)

##
user_yearbuilt = widgets.IntSlider(
    value=1950,
    min=1899,
    max=2016,
    step=1,
    description='Year house was built:',
    disabled=False,
    continuous_update=False,
    orientation='horizontal',
    readout=True,
    readout_format='d',
    style=label_layout,
    layout=wide_layout
)
display(user_yearbuilt)

## Borough/zip code dropdown
borough_zipcode_array = []
for line in borough1_zipcodes:
    #output example: ("Manhattan - zipcode", zipcode)
    borough_zipcode_array.append(
        tuple(("Manhattan - " + str(line), str(line)))
    )
for line in borough2_zipcodes:
    borough_zipcode_array.append(
        tuple(("Bronx - " + str(line), str(line)))
    )
for line in borough3_zipcodes:
    borough_zipcode_array.append(
        tuple(("Brooklyn - " + str(line), str(line)))
    )
for line in borough4_zipcodes:
    borough_zipcode_array.append(
        tuple(("Queens - " + str(line), str(line)))
    )
for line in borough5_zipcodes:
    borough_zipcode_array.append(
        tuple(("Staten Island - " + str(line), str(line)))
    )

user_borough = 1
user_zipcode = widgets.Dropdown(
    options=borough_zipcode_array,#[('Manhattan', 1), ('Bronx', 2), ('Brooklyn', 3), ('Queens', 4), ('Staten Island', 5)],

    description='Location:',
    style=label_layout,
    layout=med_layout
)
display(user_zipcode)


## size
user_land_sqrft= widgets.BoundedIntText(
    value=1975,
    min=0,
    max=10000,
    step=1,
    description='Land sq.ft.',
    disabled=False,
    style=label_layout,
    layout=med_layout
)
display(user_land_sqrft)
user_gross_sqrft = widgets.BoundedIntText(
    value=3855,
    min=0,
    max=10000,
    step=1,
    description='Gross sq.ft.',
    disabled=False,
    style=label_layout,
    layout=med_layout
)
display(user_gross_sqrft)

## Output text
final_output = widgets.HTML(value = f'<p style="font-size:2em">Click "Generate report" to get an estimate for a residence price.</p>')

display(final_output)
## Generate button
gen_button = widgets.Button(
    description='Generate report',
    disabled=False,
    button_style='success', # 'success', 'info', 'warning', 'danger' or ''
    tooltip='Click me',
    icon='' # (FontAwesome names without the `fa-` prefix)
)
gen_output = widgets.Output()

display(gen_button, gen_output)


def on_button_clicked(b):
                
    with gen_output:
        
        # get borough id from zipcode
        for index, b in enumerate(borough_list):
            if(int(user_zipcode.value) in b):
                user_borough = index + 1
        
        #predict new result
        sample = np.array([user_borough,
                           user_zipcode.value,
                           user_yearbuilt.value,
                           user_month.value,
                           user_land_sqrft.value,
                           user_gross_sqrft.value]).reshape(1, -1)
        y_pred = regressor.predict(sample)
        
        
        final_output.value = "<p style='font-size:2em'>Projected residence price: <font color='green'>$" + f"{int(y_pred):,d}</b></p>"
        print("Projected residance price: $" + f"{int(y_pred):,d}")
        

gen_button.on_click(on_button_clicked)<jupyter_output><empty_output><jupyter_text>Comparision of actual values in the dataset and predicted values using our modelGiven the same input parameters (location, month of sale, characteristics of housing unit), what is the actual price of the house and what does our model predict the price to be?<jupyter_code>pd.DataFrame({'Actual Value':y_test.reshape(-1), 'Predicted Value':y_pred.reshape(-1)}).head(n=20).style.format("{:,.0f}")<jupyter_output><empty_output><jupyter_text>Visualize the regression modelSee an example of the effect of some input parameters (location, month of sale, characteristics of housing unit) on the predicted house price using a regression scatter plot.<jupyter_code>from matplotlib.ticker import FuncFormatter
import seaborn as sns
# visualizing regression model
sns.regplot(x="GROSS SQUARE FEET", y="SALE PRICE", data=df);

plt.title('Effect of Gross size on Sale Price in Regression Model')
plt.xlabel('Gross sq.ft')
plt.ylabel('Sale Price')
plt.ticklabel_format(style='plain', axis='y')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11901886091238502, 'fraction_numerical': 0.037350181176251974, 'mean_word_length': 4.414486921529175, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '197187', 'license': 'MIT', 'n_tokens_mistral': 3972, 'n_tokens_neox': 3569, 'n_words': 877, 'path': 'ModarAljundi-app.ipynb', 'repo_name': 'ModarAljundi/CS-Capstone-NYC-App', 'size': '10763'}