<jupyter_start><jupyter_text>Observations: 1. These variables have very different scales. We will discuss this later in this chapter when we explore feature scaling.2. Many histograms are right skewed: they extend much farther to the right of the median than to the left. This may make it a bit harder for some Machine Learning algorithms to detect patterns. We will try transforming these attributes later on to have more bell-shaped distributions.<jupyter_code>sf.plot(kind="scatter", x="longitude", y="latitude", alpha=0.2)
plt.savefig('map1.png')
sf.plot(kind="scatter", x="longitude", y="latitude", alpha=0.4, figsize=(10,7),
    c="lastsoldprice", cmap=plt.get_cmap("jet"), colorbar=True,
    sharex=False)
plt.savefig('map2.png')<jupyter_output><empty_output><jupyter_text>Now let’s look at how much each independent variable correlates with the last sold price.<jupyter_code>corr_matrix = sf.corr()
corr_matrix["lastsoldprice"].sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>The last sold price tends to go up when the finished sqft and number of bathrooms go up. You can see a small negative correlation between the year built and the last sold price. Finally, coefficients close to zero mean that there is no linear correlation.<jupyter_code>from pandas.tools.plotting import scatter_matrix

attributes = ["lastsoldprice", "finishedsqft", "bathrooms", "zindexvalue"]
scatter_matrix(sf[attributes], figsize=(12, 8))
plt.savefig('matrix.png')<jupyter_output>C:\Users\Susan\Anaconda3\lib\site-packages\ipykernel\__main__.py:4: FutureWarning: 'pandas.tools.plotting.scatter_matrix' is deprecated, import 'pandas.plotting.scatter_matrix' instead.<jupyter_text>The most promising variable to predict the sold price is the zestimate, so let’s zoom in on their correlation scatterplot<jupyter_code>sf.plot(kind="scatter", x="finishedsqft", y="lastsoldprice", alpha=0.5)
plt.savefig('scatter.png')<jupyter_output><empty_output><jupyter_text>First, the correlation is indeed very strong; you can clearly see the upward trend and the points are not too dispersed. Because each neighborhood has the different home price, what we really want is the price per squre foot. So, we add a new variable "price_per_sqft".<jupyter_code>sf['price_per_sqft'] = sf['lastsoldprice']/sf['finishedsqft']
corr_matrix = sf.corr()
corr_matrix["lastsoldprice"].sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>Unfortunately, the new price_per_sqft variable has only a very smalll positive correlation with the last sold price. But we still need this varaible for grouping neighborhood. Also there are 71 neighborhoods in the data, we are going to group them.<jupyter_code>len(sf['neighborhood'].value_counts())
freq = sf.groupby('neighborhood').count()['address']
#mean_price: average price per sqft
mean = sf.groupby('neighborhood').mean()['price_per_sqft']
cluster = pd.concat([freq, mean], axis=1)
cluster['neighborhood'] = cluster.index
cluster.columns = ['freq', 'price_per_sqft','neighborhood']
cluster.describe()<jupyter_output><empty_output><jupyter_text>Cluster the neighborhood into three groups: 1. low price, 2. high price low frequency, 3. high price high frequency<jupyter_code>cluster1 = cluster[cluster.price_per_sqft < 756]
cluster1.index
cluster_temp = cluster[cluster.price_per_sqft >= 756]
cluster2 = cluster_temp[cluster_temp.freq <123]
cluster2.index
cluster3 = cluster_temp[cluster_temp.freq >=123]
cluster3.index<jupyter_output><empty_output><jupyter_text>Add group column based on the clusters<jupyter_code>def get_group(x):
    if x in cluster1.index:
        return 'low_price'
    elif x in cluster2.index:
        return 'high_price_low_freq'
    else:
        return 'high_price_high_freq'
sf['group'] = sf.neighborhood.apply(get_group)<jupyter_output><empty_output><jupyter_text>Now let's preprocess the categorical input feature, usecode<jupyter_code>sf.head()
sf.drop(sf.columns[[0, 4, 6, 7, 8, 13]], axis=1, inplace=True)
sf.head()
sf = sf[['bathrooms', 'bedrooms', 'finishedsqft', 'totalrooms', 'usecode', 'yearbuilt', 
         'zindexvalue', 'group', 'lastsoldprice']]
sf.head()<jupyter_output><empty_output><jupyter_text>Create dummy variables for two categorical variables.<jupyter_code>X = sf[['bathrooms', 'bedrooms', 'finishedsqft', 'totalrooms', 'usecode', 'yearbuilt', 
         'zindexvalue', 'group']]
Y = sf['lastsoldprice']
#create dummy variables for groups
n = pd.get_dummies(sf.group)
X = pd.concat([X, n], axis=1)
m = pd.get_dummies(sf.usecode)
X = pd.concat([X, m], axis=1)
drops = ['group', 'usecode']
X.drop(drops, inplace=True, axis=1)
X.head()
Y.shape<jupyter_output><empty_output><jupyter_text>Standardization of the data<jupyter_code>from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X = scaler.fit_transform(X)
from sklearn.cross_validation import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.3, random_state=0)
from sklearn.linear_model import LinearRegression
regressor = LinearRegression()
regressor.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Done! We now have a working Linear Regression model.<jupyter_code>y_pred = regressor.predict(X_test)
print('Liner Regression R squared: %.4f' % regressor.score(X_test, y_test))<jupyter_output>Liner Regression R squared: 0.5619<jupyter_text>So, in our model, 56.19% of the variability in Y can be explained using X. It is not so exciting. Calculate root-mean-square error (RMSE)<jupyter_code>import numpy as np
from sklearn.metrics import mean_squared_error
lin_mse = mean_squared_error(y_pred, y_test)
lin_rmse = np.sqrt(lin_mse)
print('Liner Regression RMSE: %.4f' % lin_rmse)<jupyter_output>Liner Regression RMSE: 616071.5748<jupyter_text>Our model was able to predict the value of every house in the test set within $616071 of the real price. Calculate mean absolute error (MAE)<jupyter_code>from sklearn.metrics import mean_absolute_error

lin_mae = mean_absolute_error(y_pred, y_test)
print('Liner Regression MAE: %.4f' % lin_mae)<jupyter_output>Liner Regression MAE: 363742.1631<jupyter_text>let’s try a more complex model to see whether results can be improved - the RandomForestRegressor<jupyter_code>from sklearn.ensemble import RandomForestRegressor

forest_reg = RandomForestRegressor(random_state=42)
forest_reg.fit(X_train, y_train)
print('Random Forest R squared": %.4f' % forest_reg.score(X_test, y_test))
y_pred = forest_reg.predict(X_test)
forest_mse = mean_squared_error(y_pred, y_test)
forest_rmse = np.sqrt(forest_mse)
print('Random Forest RMSE: %.4f' % forest_rmse)<jupyter_output>Random Forest RMSE: 551406.0926<jupyter_text>Much better! Let's try one more.<jupyter_code>from sklearn import ensemble
from sklearn.ensemble import GradientBoostingRegressor
model = ensemble.GradientBoostingRegressor()
model.fit(X_train, y_train)
print('Gradient Boosting R squared": %.4f' % model.score(X_test, y_test))
y_pred = model.predict(X_test)
model_mse = mean_squared_error(y_pred, y_test)
model_rmse = np.sqrt(model_mse)
print('Gradient Boosting RMSE: %.4f' % model_rmse)<jupyter_output>Gradient Boosting RMSE: 541503.7962<jupyter_text>I would consider this is our final model, let's print out feature importance<jupyter_code>feature_labels = np.array(['bathrooms', 'bedrooms', 'finishedsqft', 'totalrooms', 'yearbuilt', 'zindexvalue', 
                           'high_price_high_freq', 'high_price_low_freq', 'low_price', 'Apartment', 'Condominium', 'Cooperative', 
                          'Duplex', 'Miscellaneous', 'Mobile', 'MultiFamily2To4', 'MultiFamily5Plus', 'SingleFamily', 
                           'Townhouse'])
importance = model.feature_importances_
feature_indexes_by_importance = importance.argsort()
for index in feature_indexes_by_importance:
    print('{}-{:.2f}%'.format(feature_labels[index], (importance[index] *100.0)))<jupyter_output>Apartment-0.00%
MultiFamily5Plus-0.00%
Mobile-0.00%
Miscellaneous-0.00%
Cooperative-0.00%
Townhouse-0.00%
Condominium-0.54%
high_price_low_freq-0.69%
Duplex-1.21%
MultiFamily2To4-1.68%
high_price_high_freq-3.15%
bedrooms-3.17%
low_price-3.57%
SingleFamily-4.12%
yearbuilt-8.84%
totalrooms-9.13%
bathrooms-14.92%
zindexvalue-16.03%
finishedsqft-32.94%
{'fraction_non_alphanumeric': 0.11420950533462658, 'fraction_numerical': 0.0233996120271581, 'mean_word_length': 6.66635687732342, 'pattern_counts': {'":': 2, '<': 64, '<?xml version=': 0, '>': 64, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '524095', 'license': 'MIT', 'n_tokens_mistral': 2952, 'n_tokens_neox': 2757, 'n_words': 689, 'path': 'exercises/exercise 2 - Bay area home pricing/solutions/Predict_Bay_Area_Home_Price.ipynb', 'repo_name': 'tommyhetterscheid17/data-analytics-and-science', 'size': '8248'}