<jupyter_start><jupyter_text>Data Prep of Chicago Food Inspections DataThis notebook reads in the food inspections dataset containing records of food inspections in Chicago since 2010.  This dataset is freely available through healthdata.gov, but must be provided with the odbl license linked below and provided within this repository.  This notebook prepares the data for statistical analysis and modeling by creating features from categorical variables and enforcing a prevalence threshold for these categories.  Note that in this way, rare features are not analyzed or used to create a model (to encourage generalizability), though the code is designed so that it would be easy to change or eliminate the prevalence threshold to run downstream analysis with a different feature set. References- Data Source: https://healthdata.gov/dataset/food-inspections- License: http://opendefinition.org/licenses/odc-odbl/  Set Global Seed<jupyter_code>SEED = 666<jupyter_output><empty_output><jupyter_text>Imports<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>Read Chicago Food Inspections DataCount records and columns.<jupyter_code>food_inspections_df = pd.read_csv('../data/Food_Inspections.gz', compression='gzip')
food_inspections_df.shape<jupyter_output><empty_output><jupyter_text>Rename Columns<jupyter_code>food_inspections_df.columns.tolist()
columns = ['inspection_id', 'dba_name', 'aka_name', 'license_number', 'facility_type',
           'risk', 'address', 'city', 'state', 'zip', 'inspection_date', 'inspection_type',
           'result', 'violation', 'latitude', 'longitude', 'location']
food_inspections_df.columns = columns<jupyter_output><empty_output><jupyter_text>Convert Zip Code to StringAnd take only the first five digits, chopping off the decimal from reading the column as a float.<jupyter_code>food_inspections_df['zip'] = food_inspections_df['zip'].astype(str).apply(lambda x: x.split('.')[0])<jupyter_output><empty_output><jupyter_text>Normalize Casing of ChicagoAccept only proper spellings of the word Chicago with mixed casing accepted.<jupyter_code>food_inspections_df['city'] = food_inspections_df['city'].apply(lambda x: 'CHICAGO'
                                                                if str(x).upper() == 'CHICAGO'
                                                                else x)<jupyter_output><empty_output><jupyter_text>Filter for Facilities in Chicago Illinois<jupyter_code>loc_condition = (food_inspections_df['city'] == 'CHICAGO') & (food_inspections_df['state'] == 'IL')<jupyter_output><empty_output><jupyter_text>Drop Redundant Information- Only Chicago is considered- Only Illinois is considered- Location is encoded as separate latitute and longitude columns<jupyter_code>food_inspections_df = food_inspections_df[loc_condition].drop(['city', 'state', 'location'], 1)
food_inspections_df.shape<jupyter_output><empty_output><jupyter_text>Create Codes Corresponding to Each Violation Type by Parsing Violation Text<jupyter_code>def create_violation_code(violation_text):
    
    if violation_text != violation_text:
        return -1
    else:
        return int(violation_text.split('.')[0])
food_inspections_df['violation_code'] = food_inspections_df['violation'].apply(create_violation_code)<jupyter_output><empty_output><jupyter_text>Create Attribute Dataframes with the Unique Inspection ID for Lookups if Needed- Names- Licenses- Locations- Violations- Dates<jupyter_code>names = ['inspection_id', 'dba_name', 'aka_name']
names_df = food_inspections_df[names]
licenses = ['inspection_id', 'license_number']
licenses_df = food_inspections_df[licenses]
locations = ['inspection_id', 'address', 'latitude', 'longitude']
locations_df = food_inspections_df[locations]
violations = ['inspection_id', 'violation', 'violation_code']
violations_df = food_inspections_df[violations]
dates = ['inspection_id', 'inspection_date']
dates_df = food_inspections_df[dates]<jupyter_output><empty_output><jupyter_text>Drop Features Not Used in Statistical AnalysisFeatures such as:- `DBA Name`- `AKA Name`- `License `- `Address`- `Violations`- `Inspection Date`May be examined following statistical analysis by joining on `Inspection ID`.  **Note:** future iterations of this work may wish to consider:- Text from the the facility name- Street level information from the facility address- Prior inspections of the same facility by performing a temporal analysis of the data using `Inspection Date`<jupyter_code>not_considered = ['dba_name', 'aka_name', 'license_number', 'address', 'violation', 'inspection_date']
food_inspections_df = food_inspections_df.drop(not_considered, 1)<jupyter_output><empty_output><jupyter_text>Create Dataframes of Count and Prevalence for Categorical Features- Facility types- Violation codes- Zip codes- Inspection types<jupyter_code>facilities = food_inspections_df['facility_type'].value_counts()
facilities_df = pd.DataFrame({'facility_type':facilities.index, 'count':facilities.values})
facilities_df['prevalence'] = facilities_df['count'] / food_inspections_df.shape[0]
facilities_df.nlargest(10, 'count')
facilities_df.nsmallest(10, 'count')
violations = food_inspections_df['violation_code'].value_counts()
violations_df = pd.DataFrame({'violation_code':violations.index, 'count':violations.values})
violations_df['prevalence'] = violations_df['count'] / food_inspections_df.shape[0]
violations_df.nlargest(10, 'count')
violations_df.nsmallest(10, 'count')
zips = food_inspections_df['zip'].value_counts()
zips_df = pd.DataFrame({'zip':zips.index, 'count':zips.values})
zips_df['prevalence'] = zips_df['count'] / food_inspections_df.shape[0]
zips_df.nlargest(10, 'count')
zips_df.nsmallest(10, 'count')
inspections = food_inspections_df['inspection_type'].value_counts()
inspections_df = pd.DataFrame({'inspection_type':inspections.index, 'count':inspections.values})
inspections_df['prevalence'] = inspections_df['count'] / food_inspections_df.shape[0]
inspections_df.nlargest(10, 'count')
inspections_df.nsmallest(10, 'count')
results = food_inspections_df['result'].value_counts()
results_df = pd.DataFrame({'result':results.index, 'count':results.values})
results_df['prevalence'] = results_df['count'] / food_inspections_df.shape[0]
results_df.nlargest(10, 'count')<jupyter_output><empty_output><jupyter_text>Drop Violation Code for NowWe can join back using the Inspection ID to learn about types of violations, but we don't want to use any information about the violation itself to predict if a food inspection will pass or fail.<jupyter_code>food_inspections_df = food_inspections_df.drop('violation_code', 1)<jupyter_output><empty_output><jupyter_text>Create Risk Group FeatureIf the feature cannot be found in the middle of the text string as a value 1-3, return -1.<jupyter_code>def create_risk_groups(risk_text):
    
    try: 
        risk = int(risk_text.split(' ')[1])
        return risk
    except:
        return -1
food_inspections_df['risk'] = food_inspections_df['risk'].apply(create_risk_groups)<jupyter_output><empty_output><jupyter_text>Format Result- Encode Pass and Pass w/ Conditions as 0- Encode Fail as 1- Encode all others as -1 and filter out these results<jupyter_code>def format_results(result):
    
    if result == 'Pass':
        return 0
    elif result == 'Pass w/ Conditions':
        return 0
    elif result == 'Fail':
        return 1
    else:
        return -1
food_inspections_df['result'] = food_inspections_df['result'].apply(format_results)
food_inspections_df = food_inspections_df[food_inspections_df['result'] != -1]
food_inspections_df.shape<jupyter_output><empty_output><jupyter_text>Filter for Categorical Features that Pass some Prevalence ThresholdThis way we only consider fairly common attributes of historical food establishments and inspections so that our analysis will generalize to new establishments and inspections.  **Note:** the prevalence threshold is set to **0.1%**.<jupyter_code>categorical_features = ['facility_type', 'zip', 'inspection_type']
def prev_filter(df, feature, prevalence='prevalence', prevalence_threshold=0.001):
    
    return df[df[prevalence] > prevalence_threshold][feature].tolist()
feature_dict = dict(zip(categorical_features, [prev_filter(facilities_df, 'facility_type'),
                                               prev_filter(zips_df, 'zip'),
                                               prev_filter(inspections_df, 'inspection_type')]))<jupyter_output><empty_output><jupyter_text>Encode Rare Features with the 'DROP' String, to be Removed LaterNote that by mapping all rare features to the 'DROP' attribute, we avoid having to one-hot-encode all rare features and then drop them after the fact.  That would create an unnecessarily large feature matrix.  Instead we one-hot encode features passing the prevalence threshold and then drop all rare features that were tagged with the 'DROP' string.<jupyter_code>for feature in categorical_features:
    food_inspections_df[feature] = food_inspections_df[feature].apply(lambda x: 
                                                                      x if x in feature_dict[feature]
                                                                      else 'DROP')
feature_df = pd.get_dummies(food_inspections_df,
                            prefix=['{}'.format(feature) for feature in categorical_features],
                            columns=categorical_features)
feature_df = feature_df[[col for col in feature_df.columns if 'DROP' not in col]]
feature_df.shape<jupyter_output><empty_output><jupyter_text>Drop Features with:- Risk level not recorded as 1, 2, or 3- Result not recorded as Pass, Pass w/ Conditions, or Fail- NA values (Some latitudes and longitudes are NA)<jupyter_code>feature_df = feature_df[feature_df['risk'] != -1]
feature_df = feature_df[feature_df['result'] != -1]
feature_df = feature_df.dropna()
feature_df.shape<jupyter_output><empty_output><jupyter_text>Write the Feature Set to a Compressed CSV File to Load for Modeling and Analysis<jupyter_code>feature_df.to_csv('../data/Food_Inspection_Features.gz', compression='gzip', index=False)<jupyter_output><empty_output><jupyter_text>Write off Zip Codes to Join with Census Data<jupyter_code>zips_df.to_csv('../data/Zips.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10081505918882204, 'fraction_numerical': 0.005821851348728896, 'mean_word_length': 5.421806853582554, 'pattern_counts': {'":': 0, '<': 81, '<?xml version=': 0, '>': 82, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '12025', 'license': 'MIT', 'n_tokens_mistral': 3204, 'n_tokens_neox': 3018, 'n_words': 796, 'path': 'final_project/notebooks/01_food_inspections_data_prep.ipynb', 'repo_name': 'jstremme/DATA512-Research', 'size': '10306'}