<jupyter_start><jupyter_text>Loading Food Safety Data<jupyter_code>filename = 'data.zip'
my_zip = zipfile.ZipFile(filename)
list_names = my_zip.namelist()
list_names
bus = pd.read_csv('data/businesses.csv',encoding = 'ISO-8859-1')
ins = pd.read_csv('data/inspections.csv')
vio = pd.read_csv('data/violations.csv')<jupyter_output><empty_output><jupyter_text>Examining the Business Data<jupyter_code># is each business id unique?
bus['business_id'].is_unique<jupyter_output><empty_output><jupyter_text>What does each record represent?Each record represents all the information about a single restaurant. Each row is equivalent to one restaurant. What is the primary key?The primary key is the business_id column since even if two restaurants shared other information (like address) they would still have unique business-id's.  Zip Codes<jupyter_code># counts for resturants in each zipcode
zip_counts = bus['postal_code'].value_counts()
zip_counts.head()
# check if all data is accounted for
print('zip_counts describes', sum(zip_counts), 'records.')
print('The original data have', len(bus), 'records')
print('Missing', len(bus)-sum(zip_counts),'records')<jupyter_output>zip_counts describes 6166 records.
The original data have 6406 records
Missing 240 records<jupyter_text>Missing zip codes aren't our only problem. There are also some records where the postal code is wrong, e.g., there are 3 'Ca' and 3 'CA' values. Additionally, there are some extended postal codes that are 9 digits long, rather than the typical 5 digits.<jupyter_code>bus['postal_code_5'] = bus['postal_code'].str[:5]
bus.head()
bus[bus['postal_code'].isnull()]<jupyter_output><empty_output><jupyter_text>Many of the records with NaN in the postal_code column also have an address listed as "OFF THE GRID". At first I thought this was some bougie San Fransisco thing where the restaurants are meant to remain secret so only the most 'worthy' know where they are. But I did some research and discovered https://offthegrid.com/ which is a collection of mobile food vendors. Now it makes sense as to why many of these records are without not only postal_code but also city, state, and phone number. Since they are mobile they are not necessarily limited to any one location. Additionally, it did not cross my mind that food trucks would have health and safety inspections, but in thinkning about it, surely they should.<jupyter_code>all_sf_zip_codes = ["94102", "94103", "94104", "94105", "94107", "94108", 
                    "94109", "94110", "94111", "94112", "94114", "94115", 
                    "94116", "94117", "94118", "94119", "94120", "94121", 
                    "94122", "94123", "94124", "94125", "94126", "94127", 
                    "94128", "94129", "94130", "94131", "94132", "94133", 
                    "94134", "94137", "94139", "94140", "94141", "94142", 
                    "94143", "94144", "94145", "94146", "94147", "94151", 
                    "94158", "94159", "94160", "94161", "94163", "94164", 
                    "94172", "94177", "94188"]
weird_zip_code_businesses = bus[~bus['postal_code_5'].isin(all_sf_zip_codes)]
weird_zip_code_businesses<jupyter_output><empty_output><jupyter_text>We can look at each of these individually and notice a few things.1. It looks as though there may have been a number of data entry erros, where singular digits are different than the zip codes found on the actual sites associated with the business2. Sometimes the company operates in San Fransisco but is actually headquartered outside of the city so they will be disregarded.<jupyter_code># keeping only the good data
bus = bus[bus['postal_code_5'].isin(all_sf_zip_codes)]
bus.head()<jupyter_output><empty_output><jupyter_text>Sampling from Business Data<jupyter_code>def sample(series, n):
    return np.random.choice(series,n,replace=False).tolist()
#gather stratified random sample of business names, stratum is postal code
rest_name_list = ([(sample(bus[bus['postal_code_5']==str(code)]['name'],1)) for code in all_sf_zip_codes
                   if code in bus['postal_code_5'].value_counts()])
bus_strat_sample = pd.Series(rest[0] for rest in rest_name_list)
bus_strat_sample.head()
#gather cluster sample of business ID, cluster is postal code,
#5 cluster per sample
good_codes = [code for code in all_sf_zip_codes if code in bus['postal_code_5'].value_counts()]
post_codes = sample(good_codes, 5)
bus_cluster_sample = bus[bus['postal_code_5'].isin(post_codes)]['business_id']
bus_cluster_sample.head()<jupyter_output><empty_output><jupyter_text>Compare the strengths and weaknesses of cluster and stratified sampling:Stratified sampling allows us to poll over every single zip code in the city (for which we have a restaurant), which allows us to form a fuller picture of the city as a whole, however by only taking a SRS of n=1 within each zipcode we do not really achieve that because it puts the full weight of a zipcode on a single restaurant. Cluster sampling allows us to (in theory) get a good representation for any 5 zip codes (in this case) because it takes every restaurant from a randomly chosen zip code for comparison. However this sample is similarly problematic when we get a zip code with few restaurants or if we get a zip code which is in San Fransisco but for which we have no restaurant data (a problem which I chose to omit by further reducing the size of `all_sf_zip_codes`)<jupyter_code>#multi-stage sample, SRS of 5 postal codes, then SRS of 1 business name per code
good_codes = [code for code in all_sf_zip_codes if code in bus['postal_code_5'].value_counts()]
post_codes = sample(good_codes, 5)
rest_name_list = ([(sample(bus[bus['postal_code_5']==str(code)]['name'],1)) for code in post_codes
                   if code in bus['postal_code_5'].value_counts()])
bus_multi_sample = pd.Series(rest[0] for rest in rest_name_list)
bus_multi_sample.head()<jupyter_output><empty_output><jupyter_text>Latitude and Longitude<jupyter_code>#missing longitude?
num_missing_longs = len(bus[bus['longitude'].isnull()])
num_missing_longs
sf_dense_zip = ["94102", "94103", "94104", "94105", "94107", "94108",
                "94109", "94110", "94111", "94112", "94114", "94115",
                "94116", "94117", "94118", "94121", "94122", "94123", 
                "94124", "94127", "94131", "94132", "94133", "94134"]
num_missing_in_each_zip = bus['postal_code_5'][bus['longitude'].isnull()][bus['postal_code_5'].isin(sf_dense_zip)].value_counts()
num_missing_in_each_zip.head()<jupyter_output><empty_output><jupyter_text>Examining the proportion of null longitudes<jupyter_code>def div(x,y):
    return x/y

count_non_null = bus['postal_code_5'][~bus['longitude'].isnull()][bus['postal_code_5'].isin(sf_dense_zip)].value_counts()
fraction_missing_df = pd.concat([count_non_null.rename('count non null'),num_missing_in_each_zip.rename('count null')],axis=1,sort=False,)
fraction_missing_df['fraction null'] = fraction_missing_df['count null'] / (fraction_missing_df['count null'] + fraction_missing_df['count non null'])
fraction_missing_df.sort_values(by=['fraction null'], ascending = False, inplace= True)
fraction_missing_df.index.name = 'postal_code_5'
fraction_missing_df.head(100)<jupyter_output><empty_output><jupyter_text>Investigate the Inspection Data examine the types of inspection<jupyter_code>ins.dtypes
ins.type.value_counts()
# fix date column / add year
ins['new_date'] = pd.to_datetime(ins['date'], format='%Y%m%d')
ins['year'] = ins['new_date'].dt.year
ins.year.value_counts()<jupyter_output><empty_output><jupyter_text>Explore Inspection Scores<jupyter_code>scores_table = x=ins['score'].value_counts(ascending=True)
ax = plt.bar(x=scores_table.index, height = scores_table.values)
plt.title('Distribution of Inspection Scores')
plt.xlabel('Score')
plt.ylabel('Count');
plt.show()<jupyter_output><empty_output><jupyter_text>The scores are heavily focused in the 90 and above range with 64.7% of the data falling in that range. The distribution also has a very long left tail; the lowest value is 48, and the range of 48 to 72 comprises only 4.14% of the data. There is a pretty strong trend upwards for this data, there are some upper 90s values we never see: 99, 97, 95 as well as 50 and 51 but since there are so few points in lower ranges, it seems more possible to have missed those points. However statistically, of the 9000+ points in the 90s to not have a single of 99, 97, or 95 seems strange.I do not know the standards for rating restaurants, but it does strike me as very odd that so many of these maintain scores higher than 90 and that the frequency of scores falls of exponentially. It could be that restaurants receiving low scores are driven out be a controlling market, but it seems more likely to me that the rating system itself is prone to bias and that these scores may not represent the quality of the restaurant as a whole.<jupyter_code>print(sum(ins['score'][ins['score']>89].value_counts())/sum(ins['score'].value_counts()))
print((sum(ins['score'][ins['score']>47].value_counts())-sum(ins['score'][ins['score']>73].value_counts()))/sum(ins['score'].value_counts()))
for val in range(49,101):
    if val not in ins.score.value_counts().index:
        print(val)<jupyter_output>0.6470257347771059
0.04141470960483758
49
50
51
95
97
99<jupyter_text>what are the resturants with the worst scores?<jupyter_code>ins_named = pd.merge(ins,bus[['business_id','name','address']],how='left',on='business_id')
ins_named.sort_values(by=['score']).head()<jupyter_output><empty_output><jupyter_text>Restaurant Ratings Over Time which restaurant has improved the most? (min 3 ratings)<jupyter_code>id_more_3 = ins_named.business_id.value_counts().loc[lambda x: x>= 3].index
ins_named_filtered = ins_named[ins_named['business_id'].isin(id_more_3)][['business_id', 'score']]
ins_named_filtered[ins_named_filtered['business_id'] == 24]['score'].values
list_scores = {bus_id:ins_named_filtered[ins_named_filtered['business_id'] == bus_id]['score'].values for bus_id in id_more_3}
diffs = {}
for bus_id in list_scores:
    diffs[bus_id] = max(list_scores[bus_id])-min(list_scores[bus_id])
max_swing = ins_named[ins_named['business_id']==max(diffs,key=diffs.get)].name.values[1]
max_swing<jupyter_output><empty_output><jupyter_text>How many times has each restaurant been investigated?<jupyter_code>ins_grouped = ins.groupby(['business_id','year'])
inspections_by_id_and_year =  ins_grouped[['business_id','year']].head().groupby(['business_id','year']).sum()
inspections_by_id_and_year['count'] = ins_grouped.size().values
inspections_by_id_and_year.head()<jupyter_output><empty_output><jupyter_text>How frequently are business investigated multiple times per year?<jupyter_code>inspections_by_id_and_year['count'].value_counts()<jupyter_output><empty_output><jupyter_text>let's compare the distributions of those resturants which are investigated multiple times per year, seeing their first score compared to their second score.<jupyter_code>review_2016 = ins[ins.year == 2016].sort_values(by='new_date').groupby(['business_id']).agg(list)
review_2016 = review_2016[review_2016['score'].map(len) == 2]
scores_pairs_by_business = pd.DataFrame(review_2016.score.values,review_2016.index).rename(columns = {0:'score_pair'})
ins2016 = ins[ins['year'] == 2016]
x= np.array([dat[0] for dat in scores_pairs_by_business.score_pair.values])
y=np.array([dat[1] for dat in scores_pairs_by_business.score_pair.values])
plt.scatter(x=x,y=y)
plt.plot([50,100],[50,100],c='r')
plt.title('First Inspection Score vs. Second Inspection Score')
plt.xlabel('First Inspection Score')
plt.ylabel('Second Inspection Score')
plt.xlim([55,100.5]);
plt.show()<jupyter_output><empty_output><jupyter_text>That doesn't show too much, we can also see the distribution in ranges in another way.<jupyter_code>diff = y-x
plt.hist(diff,bins=list(range(min(diff),max(diff),2)))
plt.title('Distribution of Score Differences')
plt.xlabel('Score Differences (Second Score - First Score)')
plt.ylabel('Count');
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11882255389718076, 'fraction_numerical': 0.04900497512437811, 'mean_word_length': 6.248197115384615, 'pattern_counts': {'":': 0, '<': 75, '<?xml version=': 0, '>': 79, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '264227', 'license': 'MIT', 'n_tokens_mistral': 4293, 'n_tokens_neox': 3823, 'n_words': 1144, 'path': 'Food Safety EDA.ipynb', 'repo_name': 'alexcpeterson/SF-Fod-Safety-EDA', 'size': '12060'}