<jupyter_start><jupyter_text>AcquireAll functions are called in `wrangle.py` file. Functions- `acquire_fha_data.py` - This function takes fha data from hud website (url, sheet_name) turns it into pandas df.    - url = https://www.hud.gov/sites/dfiles/Housing/documents/Initi_Endores_Firm%20Comm_DB_FY06_FY20_Q2.xlsx    - sheet_name = "Firm Cmtmts, Iss'd and Reiss'd"    - The dataset can be physically found at https://www.hud.gov/program_offices/housing/mfh/mfdata/mfproduction  PrepareAll functions are called in `wrangle.py` file Functions- `drop_multiple_fha_numbers` - Drops repeat entries by taking the most recent option- `snake_case_column_names` - makes column names snake case- `change_to_bool` - changes columns with [0, Y] values to bool (Y= True)- `changing_data_types` - changes dtypes for datetime and turn fha_number into an object- `make_activity_construction_bool` - makes boolean columns for is_refinance and is_new_construction- `wrangle_hud` - mother function that also removes lower outlier and does some minor cell changes<jupyter_code>df = wrangle_hud()<jupyter_output><empty_output><jupyter_text>Data Summary<jupyter_code>print (f'This data frame is {df.shape[0]} rows and {df.shape[1]} columns.')
print ('Currently, each observation is one mortgage, which will change for modelling')
print()
print()
df.info()
df.head(2)
df.fiscal_year_of_firm_commitment_activity.hist(color='green', bins=15)
plt.title('Histogram of Mortgages by Year')
df.groupby('project_city').final_mortgage_amount.count().nlargest(20).plot.barh(color='magenta')
plt.title('Number of Mortgages by City')
plt.ylabel("")
sns.boxplot(df.final_mortgage_amount)
plt.title("Outliers Skewing Mortgage amount Data")
mean_diff = df.final_mortgage_amount.mean()-df.final_mortgage_amount.median()
print(f'The difference between the mean and the median is {round(mean_diff)}')

df.final_mortgage_amount.describe()<jupyter_output>The difference between the mean and the median is 4063026<jupyter_text>Exploration  PreprocessingIn order to get our data into a useable format for modeling we decided to group the data by city and state for each fiscal year to get unique observations. Below is a brief summary of the functions found in the `preprocessing.py` script which help to restructure our data into a useable format:- `get_model_df`: This function wrangles the original data, groups the data using the city, state, and fiscal year features, and aggregates the mortgage data appropriately.- `calculate_city_state_vol_delta`: This function creates the growth rate for each unique city + state + year observation using total mortgage volume.- `calculate_city_state_qty_delta`: This function creates the growth rate for each unique city + state + year observation using the quantity of mortgages.- `calculate_evolution_index`: This function calculates the evolution index using the market volume delta feature created within using the market volume feature. The evolution index is a measure which expresses the growth of a unique city + state + year observation relative to the the overall market growth rate for the whole year.- `add_new_features`: This function calls `calculate_city_state_vol_delta`, `calculate_city_state_qty_delta`, and `calculate_evolution_index` to add new features to the modeling DataFrame.- `train_validate_test_data`: This function slipts our data into train, validate, and test for modeling.- `prep_data_for_modeling`: - `labeling_future_data`:  this function creates a label for the data based on whether the quantity of mortgages or volume of mortgages is an outlier.  If it is an outlier the label is `True` in the column `should_enter`. This column becomes the target variable in the modeling dataframes.<jupyter_code>model_df = pr.get_model_df()
print(f"""Our modeling DataFrame contains {model_df.shape[0]:,} observations & {model_df.shape[1]} features""")
model_df.head()
model_df = pr.add_new_features(model_df)
print(f"""Our modeling DataFrame now contains {model_df.shape[0]:,} observations & {model_df.shape[1]} features""")
model_df.head()<jupyter_output>Our modeling DataFrame now contains 13,507 observations & 19 features<jupyter_text>Creating Labels<jupyter_code>model_df = pr.labeling_future_data(model_df)
model_df = pr.filter_top_cities(model_df)
plt.figure(figsize=(14,4))
sns.boxplot(model_df.label_quantity_of_mortgages_pop_2y)

plt.figure(figsize=(14,4))
sns.boxplot(model_df.label_total_mortgage_volume_pop_2y, color='green')
sns.scatterplot(x='label_total_mortgage_volume_pop_2y', y='label_quantity_of_mortgages_pop_2y', data=model_df, hue='should_enter')
plt.title("Scatterplot Visualizing Markets to Enter")
plt.xlabel("")
plt.ylabel("")<jupyter_output><empty_output><jupyter_text>---  ModelingWe will be using classification algorithms to predict what markets will be hot as of 2020/2021. This will help us create recommendations for the future, so that we know what market's will be worth investing resources and labor in, and what martek's are worth ignoring.We will be likely using the following features for modeling:```pythonfeatures_for_modeling = ["quantity_of_mortgages_pop", "city_state_qty_delta_pop", "ei", "median_mortgage_amount_pop"]```Our target variable (the variable we are trying to predict, will be:```pythonlabel_feature = "should_enter"```In this case, our positive case will be `should_enter_market`. When looking at our confusion matrix, and all of it's possible outcomes, it would likely look as follows:| Matrix | Actual Positive | Actual Negative ||--------|-----------------|-----------------|| Predicted Positive | `enter_market` | predicted `not_enter_market`, but really it was a hot market and a missed opportunity | | Predicted Negative | predicted `enter_market`, but really it was a cold market, and not worth investing | `not_enter_market`Traditionally, for a project like this one, we would have focus on reducing the number of `False_Positives`, because it would be far more expensive to the stakeholder if we predicted a city was going to be hot, they spend time and money, and their investment is not returned. However, because TestFit's business strategy and software deployment are all done online, with very little investment needed for traveling. This means that actually investing in a city is not costly at all. As such, we will optimize our models to reduce the number of `False_Negtives`, because we want to make sure we are not missing any potential markets that can be considered `hot markets` in 2020 and 2021.Given that we have a low number of `positive` labels in our data, we will have to do something called **Oversampling**. This is a practice use in the field to basically help the predictive model by calling attention to the postiive labels and their patterns. We will create duplicate positive values, so that the model becomes more effective at predicting these values.  PreprocessingWe want to get the data from zero, as to reduce the risk of any information spillage.<jupyter_code>features_for_modeling = ["quantity_of_mortgages_pop", "city_state_qty_delta_pop", "ei", "median_mortgage_amount_pop"]
label_feature = "should_enter"
train_scaled, validate_scaled, test_scaled, y_train, y_validate, y_test = preprocessing_main_function(features_for_modeling, label_feature)<jupyter_output><empty_output><jupyter_text>This is how the data will look like before modeling:<jupyter_code>train_scaled<jupyter_output><empty_output><jupyter_text>Decision Tree<jupyter_code>features_for_modeling = ["quantity_of_mortgages_pop", "city_state_qty_delta_pop", "ei", "median_mortgage_amount_pop"]
label_feature = "should_enter"
train_scaled, validate_scaled, test_scaled, y_train, y_validate, y_test = preprocessing_main_function(features_for_modeling, label_feature)
predictions = pd.DataFrame({"actual": y_train, "baseline": y_train.mode()[0]})
for i in range(1, 20):
    clf, y_pred = model.run_clf(train_scaled, y_train, i)
    score = clf.score(train_scaled, y_train)
    validate_score = clf.score(validate_scaled, y_validate)
    _, _, report = model.accuracy_report(clf, y_pred, y_train)
    recall_score = report["True"].recall
    print(f"Max_depth = {i}, accuracy_score = {score:.2f}. validate_score = {validate_score:.2f}, recall = {recall_score:.2f}")
clf, y_pred = model.run_clf(train_scaled, y_train, 4)
predictions["decision_tree"] = y_pred
accuracy_score, matrix, report = model.accuracy_report(clf, y_pred, y_train)
print(accuracy_score)
print(matrix)
report
coef = clf.feature_importances_
# We want to check that the coef array has the same number of items as there are features in our X_train dataframe.
assert(len(coef) == train_scaled.shape[1])
coef = clf.feature_importances_
columns = train_scaled.columns
df = pd.DataFrame({"feature": columns,
                   "feature_importance": coef,
                  })
df = df.sort_values(by="feature_importance", ascending=False)
sns.barplot(data=df, x="feature_importance", y="feature", palette="Blues_d")
plt.title("What are the most influencial features?")<jupyter_output><empty_output><jupyter_text>Interestingly, it seems that when it comes to decision tree, the `evolution_index` is actually the most indicative feature, along side the change in number of mortgage's approved. The total `quantity_of_mortgages_pop` doesn't seem to be as influencial in the predictions.  Random Forest<jupyter_code>features_for_modeling = ["quantity_of_mortgages_pop", "city_state_qty_delta_pop", "ei", "median_mortgage_amount_pop"]
label_feature = "should_enter"
train_scaled, validate_scaled, test_scaled, y_train, y_validate, y_test = preprocessing_main_function(features_for_modeling, label_feature)
for i in range(1, 20):
    rf, y_pred = model.run_rf(train_scaled, y_train, 1, i)
    score = rf.score(train_scaled, y_train)
    validate_score = rf.score(validate_scaled, y_validate)
    _, _, report = model.accuracy_report(clf, y_pred, y_train)
    recall_score = report["True"].recall
    print(f"Max_depth = {i}, accuracy_score = {score:.2f}. validate_score = {validate_score:.2f}, recall = {recall_score:.2f}")
rf, y_pred = model.run_rf(train_scaled, y_train, 1, 3)
predictions["random_forest"] = y_pred
accuracy_score, matrix, report = model.accuracy_report(rf, y_pred, y_train)
print(accuracy_score)
print(matrix)
report
coef = rf.feature_importances_
columns = train_scaled.columns
df = pd.DataFrame({"feature": columns,
                   "feature_importance": coef,
                  })
df = df.sort_values(by="feature_importance", ascending=False)
sns.barplot(data=df, x="feature_importance", y="feature", palette="Blues_d")
plt.title("What are the most influencial features?")<jupyter_output><empty_output><jupyter_text>Interestingly, for the random_forest model, the delta of the number of loans approved by city where the most important or influencial indicator of whether a city would be `a hot martket` or not. The evolution index was the second most influencial feature. Again, the total `quantity_of_morgages_pop` was the least influencial feature.  KNN<jupyter_code>features_for_modeling = ["quantity_of_mortgages_pop", "city_state_qty_delta_pop", "ei", "median_mortgage_amount_pop"]
label_feature = "should_enter"
train_scaled, validate_scaled, test_scaled, y_train, y_validate, y_test = preprocessing_main_function(features_for_modeling, label_feature)
for i in range(1, 20):
    knn, y_pred = model.run_knn(train_scaled, y_train, i)
    score = knn.score(train_scaled, y_train)
    validate_score = knn.score(validate_scaled, y_validate)
    _, _, report = model.accuracy_report(clf, y_pred, y_train)
    recall_score = report["True"].recall
    print(f"Max_depth = {i}, accuracy_score = {score:.2f}. validate_score = {validate_score:.2f}, recall = {recall_score:.2f}")
knn, y_pred = model.run_knn(train_scaled, y_train, 2)
predictions["knn"] = y_pred
accuracy_score, matrix, report = model.accuracy_report(knn, y_pred, y_train)
print(accuracy_score)
print(matrix)
report
# How do the different models compare on accuracy?
print("Accuracy Scores")
print("---------------")
for i in range(predictions.shape[1]):
    report = model.create_report(predictions.actual, predictions.iloc[:,i])
    print(f'{predictions.columns[i].title()} = {report.accuracy[0]:.2f}')
# How do the different models compare on recall?
print("Recall Scores")
print("---------------")
for i in range(predictions.shape[1]):
    report = model.create_report(predictions.actual, predictions.iloc[:,i])
    print(f'{predictions.columns[i].title()} = {report["True"].loc["recall"]:.2f}')
# How do the different models compare on recall?
print("Precision Scores")
print("---------------")
for i in range(predictions.shape[1]):
    report = model.create_report(predictions.actual, predictions.iloc[:,i])
    print(f'{predictions.columns[i].title()} = {report["True"].loc["precision"]:.2f}')<jupyter_output>Precision Scores
---------------
Actual = 1.00
Baseline = 0.59
Decision_Tree = 0.76
Random_Forest = 0.78
Knn = 1.00<jupyter_text>Conclusion:Overall, we see that because we have optimized for *recall*, the accuracy scores are a bit lower than expected. However, our recall scores are really good. We will choose the KNN model as the most effective model, given that it consistently achieved the best scores (for accuracy, recall and precision).   Evaluate<jupyter_code>rf, y_pred = model.run_rf(train_scaled, y_train, 1, 3)
y_pred = rf.predict(test_scaled)
accuracy_score, matrix, report = model.accuracy_report(rf, y_pred, y_test)
print(accuracy_score)
print(matrix)
report<jupyter_output>Accuracy on dataset: 0.75
       False  True 
False     45     27
True      16     86<jupyter_text>---<jupyter_code>knn, y_pred = model.run_knn(train_scaled, y_train, 2)
y_pred = knn.predict(test_scaled)
accuracy_score, matrix, report = model.accuracy_report(knn, y_pred, y_test)
print(accuracy_score)
print(matrix)
report<jupyter_output>Accuracy on dataset: 0.90
       False  True 
False     55     17
True       0    102<jupyter_text>---- Prediction<jupyter_code>df = pr.get_model_df()
df = pr.add_new_features(df)
df = pr.filter_top_cities(df)
df.head()
features_for_predicting = ["quantity_of_mortgages_pop", "city_state_qty_delta_pop", "ei", "median_mortgage_amount_pop"]
predictions = df[(df.year == 2020) | (df.year == 2019)].groupby("city_state")[features_for_predicting].mean()
predictions
# Helper function used to updated the scaled arrays and transform them into usable dataframes
def return_values_prediction(scaler, df):
    train_scaled = pd.DataFrame(scaler.transform(df), columns=df.columns.values).set_index([df.index.values])
    return scaler, train_scaled

# Linear scaler
def min_max_scaler_prediction(df):
    scaler = MinMaxScaler().fit(df)
    scaler, df_scaled = return_values_prediction(scaler, df)
    return scaler, df_scaled
scaler, predictions_scaled = min_max_scaler_prediction(predictions)
predictions["label"] = rf.predict(predictions_scaled)
predictions
city = predictions.reset_index().city_state.str.split("_", n=1, expand=True)[0]

state = predictions.reset_index().city_state.str.split("_", n=1, expand=True)[1]
predictions = predictions.reset_index()
predictions["city"] = city

predictions["state"] = state
predictions
predictions[predictions.label == True]
predictions.to_csv("predictions.csv")
plt.figure(figsize=(15,5))
ax = sns.barplot(data=predictions, x="city", y="ei", hue="label")
plt.title("What markets will look like in 2021, based on evolution index")
plt.xticks(rotation=45, ha="right")
plt.xlabel("City")
plt.ylabel("Evolution Index (%)")
new_labels = ['Markets to not enter', 'Markets to enter']
h, l = ax.get_legend_handles_labels()
ax.legend(h, new_labels)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1056791382497291, 'fraction_numerical': 0.009688316654981197, 'mean_word_length': 6.02013422818792, 'pattern_counts': {'":': 6, '<': 44, '<?xml version=': 0, '>': 44, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '617346', 'license': 'Unlicense', 'n_tokens_mistral': 5116, 'n_tokens_neox': 4860, 'n_words': 1452, 'path': 'mvp_folder/mvp_notebook.ipynb', 'repo_name': 'hud-capstone/Data-and-Urban-Development', 'size': '15689'}