<jupyter_start><jupyter_text>Relations Between restaurants scores and boroughs, cuisine types in NYC 1. Brief IntroductionMy dataset is from enigma, which is acquired from the unannounced inspections of restaurants conducted by New York City Health Department at least once a year. The inspectors check for compliance in food handling, food temperature, personal hygiene and vermin control. Each violation of a regulation gets a certain number of points. At the end of the inspection, the inspector totals the points, and this number is the restaurant's inspection score. The lower the score, the better the grade.In the dataset, the first few columns shows some basic information such as its camis ID(New York City-Wide Agency Management Information System (CAMIS) ID Number. This is a unique identifier for the entity (restaurant)), name, area located and so on. The rest columns contains information about the inspection date, violation of that restaurant, score and grade.The project is created to try to identify restaurants with good scores and try to find the relations between scores and borough or cuisine type.<jupyter_code>import pandas as pd
import seaborn as sns
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.pyplot import hist
df = pd.read_csv('https://tufts.box.com/shared/static/bumca8ezkpf7epqr956wyeuiog5eejb2.csv')
df.head()<jupyter_output><empty_output><jupyter_text>2. Data cleanClean the data, drop columns that will not be used, remove rows of scores that don't have values. In my project, I will not use the total grade to evaluate the restaurant because in this dataset, most restauratns only have scores, instead of grades.Cleaning data would be helpful when picking the data and to avoid mistakes.Drop the 'NAN' values in 'score' column<jupyter_code>df_clean = df.drop(['camis', 'building', 'street', 'zipcode', 'phone', 'inspection_date', 'action', 'violation_code', 'critical_flag', 'grade', 'grade_date', 'record_date', 'inspection_type'], axis = 1)
df_val = df_clean.loc[df_clean['score'].dropna().index]
df_val.head()<jupyter_output><empty_output><jupyter_text>3. Find the score range and prepare for plotIt is necessary to know the score range in the dataset in order to make a well-shaped plot.Since the maximum score is 156 and the minimun score is -1, I set three standards for the score, which are less than 50, between 50 and 100, more than 100. As the smaller the score is, the better the restaurants are, I plot the number of restaurants in each cuisine type to compare.<jupyter_code>print('biggest score =', df_val['score'].max(), ', smallest score =', df_val['score'].min())
#List all boroughs and pass them to array "boro_array"
boro_array = df_val['boro'].unique()
boro_array
#get the dataframe of each borough individually
df_brooklyn = df_val[df_val['boro'] == 'BROOKLYN']
df_queens = df_val[df_val['boro'] == 'QUEENS']
df_sta_island = df_val[df_val['boro'] == 'STATEN ISLAND']
df_manhattan = df_val[df_val['boro'] == 'MANHATTAN']
df_bronx = df_val[df_val['boro'] == 'BRONX']
print('Number of restaurants in every borough:\nBrooklyn:', len(df_brooklyn),
     '\nQueens:', len(df_queens),
     '\nStaten', len(df_sta_island),
     '\nManhattan', len(df_manhattan),
     '\nBronx', len(df_bronx))<jupyter_output>Number of restaurants in every borough:
Brooklyn: 91166 
Queens: 81072 
Staten 12598 
Manhattan 142536 
Bronx 32489<jupyter_text>4. Plot the scores related to boroughs<jupyter_code>#barchart
number_restaurants = [len(df_brooklyn), len(df_queens), len(df_sta_island), len(df_manhattan), len(df_bronx)]
y_pos = [0,1.1,2.3,3.5,4.6]
plt.bar(y_pos, number_restaurants, align='center')
plt.xticks(y_pos, boro_array)
plt.title('Number of restaurants in NYC')
plt.rcParams['figure.figsize'] = (8, 5)
print(plt.show())

#piechart
df_pie = pd.DataFrame({'number of restaurants': [len(df_brooklyn), len(df_queens), len(df_sta_island), len(df_manhattan), len(df_bronx)]},
                 index=['Brooklyn', 'Queens', 'Staten Island', 'Manhattan', 'Bronx'])
pie_chart = df_pie.plot.pie(y = 'number of restaurants', figsize = (7,7), legend = False, autopct='%1.0f%%', fontsize = 15)
print(pie_chart)
#get the array of score under 50
score_under50 = [len(df_brooklyn[df_brooklyn['score'] <= 50]), len(df_queens[df_queens['score'] <= 50]),
                len(df_sta_island[df_sta_island['score'] <= 50]), len(df_manhattan[df_manhattan['score'] <= 50]),
                len(df_bronx[df_bronx['score'] <= 50])]
print('Scores in Brooklyn, Queens, Staten Island, Manhattan, Bronx:\nScores <= 50:     ', score_under50)

#get the array of score between 50 and 100
score_50_to_100 = [len(df_brooklyn[(df_brooklyn['score'] > 50) & (df_brooklyn['score'] < 100)]),
                   len(df_queens[(df_queens['score'] > 50) & (df_queens['score'] < 100)]),
                   len(df_sta_island[(df_sta_island['score'] > 50) & (df_sta_island['score'] < 100)]),
                   len(df_manhattan[(df_manhattan['score'] > 50) & (df_manhattan['score'] < 100)]),
                   len(df_bronx[(df_bronx['score'] > 50) & (df_bronx['score'] < 100)])]
print('50 < Scores < 100:', score_50_to_100)

#get the array of score over 100
score_over100 = [len(df_brooklyn[df_brooklyn['score'] >= 100]), len(df_queens[df_queens['score'] >= 100]),
                 len(df_sta_island[df_sta_island['score'] >= 100]), len(df_manhattan[df_manhattan['score'] >= 100]),
                 len(df_bronx[df_bronx['score'] >= 100])]
print('Scores >= 100:    ', score_over100)
df_sco = pd.DataFrame({'scores <= 50': score_under50, '50 < scores < 100': score_50_to_100, 'scores >= 100' : score_over100})
df_sco['Boroughs'] = boro_array[0:5]
ax = df_sco.plot(x = 'Boroughs', y = 'scores <= 50', kind = 'bar', color = 'C1')
df_sco.plot(x = 'Boroughs', y = '50 < scores < 100', kind="bar", ax=ax, color="C2")
df_sco.plot(x = 'Boroughs', y= 'scores >= 100', kind="bar", ax=ax, color="C3")
plt.show()

df_under25 = pd.DataFrame({'Scores': score_under50},
                          index = boro_array[0:5])
df_under25.plot.pie(y='Scores', figsize=(7, 7), legend = False,autopct='%1.0f%%', title = 'Percentage of scores under 50'  )<jupyter_output><empty_output><jupyter_text>5. Statistics related to cuisine types<jupyter_code>#numebr of different cuisine types in the dataset
len(df_val['cuisine_description'].value_counts())
#ranked by restaurants counts
df_val['cuisine_description'].value_counts().head(10)
cuisine_list = df_val['cuisine_description'].value_counts().head(10).index.tolist()
cuisine_mean_score = [] #creat a list to save mean score for each cuisine
cuisine_median_score = [] #save median score for each cuisine
for i in range(10):
    cuisine_mean_score.append(df_val[df_val['cuisine_description'] == cuisine_list[i]].score.mean())
    cuisine_median_score.append(df_val[df_val['cuisine_description'] == cuisine_list[i]].score.median())
print('mean score:\n', cuisine_mean_score, '\nmedian score:\n', cuisine_median_score)
y_pos = range(10)
cuisine_list[3] = 'Latin'
plt.bar(y_pos, cuisine_mean_score, align='center')
plt.xticks(y_pos, cuisine_list, fontsize = 10, rotation = 30)
plt.title('Mean score of different cuisine')
plt.rcParams['figure.figsize'] = (10, 10)
print(plt.show())
y_pos = range(10)
plt.bar(y_pos, cuisine_median_score, align='center')
plt.xticks(y_pos, cuisine_list, fontsize = 10, rotation = 30)
plt.title('Median score of different cuisine')
plt.rcParams['figure.figsize'] = (10, 10)
print(plt.show())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13504136642647452, 'fraction_numerical': 0.031491860154790496, 'mean_word_length': 5.563047285464098, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 33, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '536222', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2755, 'n_tokens_neox': 2466, 'n_words': 713, 'path': 'ChaoWuu/Project2.ipynb', 'repo_name': 'pezLyfe/applied_ds', 'size': '7494'}