<jupyter_start><jupyter_text>(Note that the race/ethnicity numbers in this output aren't the same values we used in our presentation. The presentation numbers were directly from the Census Bureau's overall race/ethnicity numbers which were calculated, for example, as "white only" as opposed to the `white_2019` feature in our dataset here which included percentages of anyone who identified as white only or white in addition to another race. The Census began including the ability to choose more than one race during the 2000 Census.)  Let's try a clustering algorithm.<jupyter_code># Checking to see which of these duplicate columns we should drop. 
# We used this cell iteratively to examine different features of the dataframe.

df[['FIPS',
    'state', 
    'county',
    'pop_2019',
    'total_pop_age_sex',
    'white_2019',
    'white_not_hispanic_2019',
    'native_2019',
    'pac_isl_2019',
    'black_2019',
    'hispanic_2019',
    'asian_2019',
    'Metro2013',
    'rural_urban_continuum_code_2013',
    'retirement_destination_2015_update'
   ]]
df_tx = df[df['state']== ' Texas']
df_tx<jupyter_output><empty_output><jupyter_text>When we initially ran the k-means clustering algorithm, we decided to throw a lot of variables at the model. These were the only ones that we initially dropped. However, the silhouette score loop algorithm kept recommending only two clusters (not useful in this context), even when we focused just on Texas. We include this code just to show the analytic and modeling process.```pythonX = df_tx.drop(columns = ['tot_pop_race_ethnicity',                      'total_pop_poverty',                      'median_household_income_2019',                      'median_individual_income_2019',                       'tot_white',                       'tot_black_af_am',                       'tot_aminalnat',                       'tot_asian',                       'tot_hawopi',                       'white_2019',                       'pop_below_poverty',                       'under_18_percent',                       'mean_hh_income',                       'total_families',                       'mean_family_income',                       'total_pop_age_sex',                       'FIPS',                       'county',  Would be "cheating" to include county and state.                      'state'                     ])``` You can see in this next cell all the columns we had to pick from initially.<jupyter_code>df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Index: 3132 entries, 01001 to 56045
Data columns (total 54 columns):
 #   Column                                 Non-Null Count  Dtype  
---  ------                                 --------------  -----  
 0   total_pop_age_sex                      3132 non-null   float64
 1   median_age                             3132 non-null   float64
 2   sex_ratio_males                        3132 non-null   float64
 3   under_18_percent                       3132 non-null   float64
 4   over_18_percent                        3132 non-null   float64
 5   over_65_percent                        3132 non-null   float64
 6   county                                 3132 non-null   object 
 7   state                                  3132 non-null   object 
 8   total_hh                               3132 non-null   float64
 9   median_hh_income                       3132 non-null   float64
 10  mean_hh_income                         3132 non-null   float64
 11  tota[...]<jupyter_text>We finally opted to just look at the following five features for our clustering algorithm.<jupyter_code>X = df[['percent_below_poverty', 
        'smoking_ban_2010', 
        'unemployment_rate_2019', 
        'average_2011_upper_bound_pct_obesity',
        'rural_urban_continuum_code_2013'
       ]]
X.info()
X['smoking_ban_2010'].value_counts()<jupyter_output><empty_output><jupyter_text>One hot encode the necessary columns (in our case, just the `smoking_ban_2010`).<jupyter_code>ohe = OneHotEncoder(sparse = False, drop = ['none'])
smart_encoder = make_column_transformer(
                (ohe, ['smoking_ban_2010']),
                remainder = 'passthrough',
                verbose_feature_names_out=False)
X_encoded = smart_encoder.fit_transform(X)
X_encoded.shape<jupyter_output><empty_output><jupyter_text>Scale the features<jupyter_code>ss = StandardScaler()
X_scaled = StandardScaler().fit_transform(X_encoded)
smart_encoder.get_feature_names_out()
X_scaled = pd.DataFrame(data = X_scaled, columns = smart_encoder.get_feature_names_out())
X_scaled.head()
silhouette_list = []
for i in range(2, 51):
    kmeans = KMeans(n_clusters = i, random_state = 10)
    kmeans.fit(X_scaled)
    silhouette_list.append(silhouette_score(X_scaled, kmeans.labels_))
plt.plot(range(2, 51)[:30], silhouette_list[:30], marker = 'o')
plt.ylabel('Silhouette Score')
plt.xlabel('Number of Clusters')
plt.title('Silhouette Score for K-means for All Counties');
kmeans_small = KMeans(n_clusters = 4, random_state = 10)
kmeans_small
kmeans_small.fit(X_scaled)
labels_small = kmeans_small.labels_
print(f'The silhouette score for KMEANS with four clusters is : {round(silhouette_score(X_scaled, kmeans_small.labels_),3)}.')
X['labels_small'] = labels_small
X
# Reset the index to get the FIPS back into the dataframe to make it export/import correctly.
X.reset_index(inplace = True)
# Export the modeled dataset in so that the `streamlit` app can graph the counties by the clustering label and use the FIPS codes.
X.to_csv(
    '../data/cleaned/kmeans_clusters_with_labels_and_features.csv',
    index = False
)
test_df = pd.read_csv(
    '../data/cleaned/kmeans_clusters_with_labels_and_features.csv',
    dtype = {'fips':object}
)
test_df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08446063128685397, 'fraction_numerical': 0.036073534512660425, 'mean_word_length': 2.548923076923077, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '381519', 'license': 'CC0-1.0', 'n_tokens_mistral': 1893, 'n_tokens_neox': 1654, 'n_words': 469, 'path': 'code/02-3-Modeling-e.ipynb', 'repo_name': 'marcjreid/climate-change-analysis-group-project', 'size': '5766'}