<jupyter_start><jupyter_text>Script to Alter Risk Indicators and Geographies    RequirementsThe Script requires two forms of data: 1. County level raw data with state abbreviation,  2. A decision matrix.    With these two forms of data, you can select your metrics, indicators, and key states.  Changing Parameters To use the function get_final_scores, you will: * update a list to include only states you care about * revise a dictionary where the KEY will be your category (or metric) and the value will be a TUPLE containing an abbreviation, and the list of the variable names to be included in your category.  By feeding the risk indicators into a dictionary form and editing which states to include (key  states), the script will populate a CSV.<jupyter_code>import os
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot  as plt
%matplotlib inline
pd.set_option('display.max_rows', 50)
pd.set_option('display.max_columns', 500)
pd.set_option('display.width', 10000)
#Read In Raw Data
all_df = pd.read_csv('all_states_risk_data.csv')
all_df=all_df.replace([np.inf, -np.inf], np.nan)
# Read In Decision Matrix
actions = pd.read_csv('action_map.csv')
actions.columns
actions.head()
#Understand  all available variables
print(all_df.columns)
all_df.shape<jupyter_output>Index(['Unnamed: 0', 'Full State', 'short_fips', 'State_Abbr', ' County Name', 'Non-white Residents', 'Senior Residents', 'Senior Poll Workers', 'Reported Difficulty ', 'In-Person Voters', 'Inactive Voters', 'Voters Per Location', 'Voters Per Machine', 'Voters Per Pollworker', 'Wait Times', 'Wait Count', '# of Same Day Locations', '# of Machines', '# of Poll Workers', '# of Senior Poll Workers', '# of Voters Removed', '# of Precincts', '#  Mail In Ballots', '% Early Vote', '%  Rejected Ballots', 'Median Age', 'Median Household Income', 'Total Population', 'Total Eligible Voters', '% Senior Poll Worker', '% Votes in Person', '% Inactive Voters', '%Nonwhite', '% Residents Over 60'], dtype='object')<jupyter_text>Customize Metrics and  Indicators in the Below ChunkDefine your metrics, indicators, and abbreviations below.<jupyter_code># Choose Metrics and Geographies   

#The key is the metric, and the indicators are what's included
metric_map = {
    'priority' : ('PRI',['Non-white Residents','Senior Residents']),
    'recruiting risk' : ('REC',['Senior Poll Workers','Reported Difficulty ']),
     'demand risk' : ('D',['In-Person Voters', 'Inactive Voters']),
    'resource supply' : ('S',['Voters Per Location','Voters Per Machine','Voters Per Pollworker','Wait Times']),
}

#Default is All States -- update list to include only relevant states using the 2 Letter state abbreviation
all_states = [state for state in all_df['State_Abbr'].unique()] # <----- default is all states
key_states = ['TX','OH', 'AZ', 'PA', 'WI', 'NC','FL', 'MI']  # <----- customize this as function input

agg_method='median'<jupyter_output><empty_output><jupyter_text>After filling this out, simply run all chunks below<jupyter_code># Automatically populates based on the dictionary
the_metrics = [i for i in metric_map.keys()]
risk_indicators = [var for k in the_metrics for var in metric_map[k][1]]
agg_metrics = ('mean', 'median', 'max', 'min')
region_ids = ['Unnamed: 0', 'Full State', 'short_fips', 'State_Abbr', ' County Name']
#Generating these percentiles for plug and play
other_vars = [j for j in all_df.columns if (j not in risk_indicators) & (j not in region_ids)]
#other_vars
print('The metrics we are looking at are:', the_metrics)
print('The indicators that compose the metrics are:', risk_indicators)
print('The geographiesbeing looked at are:', key_states)
def get_percentiles(df, key_states):
    df = df[df['State_Abbr'].isin(key_states)]
    for i in metric_map.keys():
        for j in metric_map[i][1]:
            keep_series=df.groupby("State_Abbr")[j].count() >0
            viable_states = keep_series[keep_series==True].index
            temp_df = df[df['State_Abbr'].isin(viable_states)]
            intermed= temp_df.groupby("State_Abbr")[j].rank(pct=True, na_option="keep")
            df["("+metric_map[i][0]+") "+j] = intermed
    for other_var in other_vars:
        keep_series=df.groupby("State_Abbr")[other_var].count() >0
        viable_states = keep_series[keep_series==True].index
        temp_df = df[df['State_Abbr'].isin(viable_states)]
        intermed= temp_df.groupby("State_Abbr")[other_var].rank(pct=True, na_option="keep")
        df["(O) "+other_var.title()] = intermed
    return df
        
def get_agg_columns(df, agg_method="median"):
    funct_map = {'median': np.nanmedian,
            'mean': np.nanmean, 
            'min': np.min,
            'max': np.max}
    for metric in metric_map.keys():
        percent_list = ["("+metric_map[metric][0]+") " + j for j in metric_map[metric][1]] 
        df[metric.title()+" ("+agg_method.title()+")"] = funct_map[agg_method](df[percent_list], axis=1)
        if agg_method not in funct_map.keys():
            print('Please select an acceptable metric: "min", "max", "mean", or "median".')
    return df

def get_final_scores(df):
    for i in metric_map.keys():
        keep_series=df.groupby("State_Abbr")[i.title()+" ("+agg_method.title()+")"].count() >0
        viable_states = keep_series[keep_series==True].index
        temp_df = df[df['State_Abbr'].isin(viable_states)]
        rel_scores = temp_df.groupby("State_Abbr")[i.title()+" ("+agg_method.title()+")"].rank(pct=True, na_option="keep")
        rel_heat = rel_scores.apply(lambda x: "High" if x > 0.75  else ("Low" if x <0.5 else (np.nan if np.isnan(x) else "Medium")))
        df[i.title()+" (Final)"] = rel_scores
        df[i.title()+" (LMH)"]= rel_heat
    return df

def get_final_df(df, action_df, metric_map, key_states, agg_method='median'):
    agg_method = agg_method
#     func_list = [get_percentiles, get_agg_columns, get_final_scores]
#     for i in func_list:
#         df = i(df)
    df = get_percentiles(df,key_states)
    df = get_agg_columns(df,agg_method)
    df = get_final_scores(df)
    final_metrics = [metric for metric in df.columns if "(LMH)" in metric]
    df[final_metrics] = df[final_metrics].fillna("Unknown")
    df['concat']=  df[final_metrics].agg("".join, axis=1)
    final_df = pd.merge(df,action_df[['concat', 'Action']], on='concat', how='left')
    return final_df

scored_df = get_final_df(all_df, actions, metric_map, all_states)
#Test with Arizona
scored_df[scored_df['State_Abbr']=="AZ"].head()<jupyter_output><empty_output><jupyter_text>The End :)<jupyter_code># Write Output
scored_df.to_csv('all_states_risk_scores.csv')
scored_df[scored_df['State_Abbr'].isin(key_states)].to_csv('key_states_risk_scores.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14099332839140105, 'fraction_numerical': 0.004744255003706449, 'mean_word_length': 4.751065643648764, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '475429', 'license': 'MIT', 'n_tokens_mistral': 2300, 'n_tokens_neox': 2214, 'n_words': 622, 'path': 'data/risk_scoring_for_toolkit_public.ipynb', 'repo_name': 'dssg/voter-protection', 'size': '6745'}