<jupyter_start><jupyter_text>WORKSTATION<jupyter_code>beats.plot()
beats.head()
cleaned_beats = beats[['beat','name','geometry']]
merged_beat = pd.merge(cleaned_beats,beat_ik,left_on='beat'
                       ,right_on='police_beat')[['geometry','police_beat','injured','killed','name']]
merged_beat.sort_values('injured',ascending=False).head()
fig, ax = plt.subplots(figsize=(20, 20))
merged_beat.plot(column='injured',ax=ax, legend = True)
#fig.suptitle('Injuries in each Beat', fontsize=20);
plt.savefig('choropleth_injured.png')
merged_beat.sort_values('killed',ascending=False).head()
fig, ax = plt.subplots(figsize=(20, 20))
merged_beat.plot(column='killed',ax=ax, legend = True)
plt.savefig('choropleth_killed.png')<jupyter_output><empty_output><jupyter_text>Impact:  How important/interesting is the question being asked?Data Analysis and Visualization: Are the findings presented in an aesthetically pleasing and easy to understand manner? Is the data analysis rigorous and mathematically sound?Creativity & Innovation: Are there advanced, creative modelling and analysis techniques being used? Are there non-orthodox analysis tools being used?Presentation: How clearly are the findings presented? Have the ideas been articulated well?Vision Future plans for the project? Is there a clear idea of how they would like to use their findings?   CLEANING THE DATASETS  Traffic Count<jupyter_code>def clean_traffic_count(df,list_columns):
    for i in list_columns:
        if type(df[i][0]) == str:
            df[i] = pd.to_numeric(df[i].str.strip()).fillna(0)
        else: 
            df[i] = df[i].fillna(0)
cleaned_traffic_count = traffic_counts.drop(['id','file_no'],axis=1)
cleaned_traffic_count.dtypes
# all_count and northbound_count are objects when they should be floats; we need to convert them
traffic_count_cols = ['all_count','northbound_count','southbound_count','eastbound_count','westbound_count']
clean_traffic_count(cleaned_traffic_count,traffic_count_cols)
cleaned_traffic_count['count_date']=cleaned_traffic_count['count_date'].apply(pd.to_datetime)
cleaned_traffic_count.head()
cleaned_traffic_count['street_name'].unique()
sum(cleaned_traffic_count['total_count']-(cleaned_traffic_count[traffic_count_cols[0]]
+cleaned_traffic_count[traffic_count_cols[1]]+cleaned_traffic_count[traffic_count_cols[2]]
+cleaned_traffic_count[traffic_count_cols[3]]+cleaned_traffic_count[traffic_count_cols[4]]))
# IT WORKS! All counts add up to total counts
traffic_dict
# already pretty clean<jupyter_output><empty_output><jupyter_text>Collisions<jupyter_code>cleaned_collisions = collisions.drop('report_id',axis=1)
cleaned_collisions['hit_run_lvl'].fillna('NEITHER',inplace = True)
cleaned_collisions['date_time'] = cleaned_collisions['date_time'].apply(pd.to_datetime)
cleaned_collisions.head()
cleaned_collisions['street_type'].unique()
cleaned_collisions['hit_run_lvl'].value_counts()<jupyter_output><empty_output><jupyter_text>------------------------<jupyter_code>beats = pd.read_csv('pd_beats_dictionary_datasd.csv')
for i in beats['description']:
    print(i)
beats = gpd.read_file('pd_beats_datasd.shp')
beats.shape
beats.plot()
potentially_interesting = pd.merge(beats,cleaned_collisions,left_on='beat',right_on='police_beat')<jupyter_output><empty_output><jupyter_text>ANALYSIS QUESTIONS: For traffic count: What days are worst for traffic?Which direction is the worst?  Traffic over time<jupyter_code>roads = gpd.read_file('roads_datasd.shp')
slim_roads = roads[['geometry','rd30full']]
slim_roads['rd30full'] = slim_roads['rd30full'].str.lower()
slim_roads.head()
'''
for i in range(0,500,2):
    tester = interesting[interesting['count_date']==array_item[i]]
    array = slim_roads['rd30full'].str.contains('!') 
    for j in tester['street_name']:
        array += slim_roads['rd30full'].str.contains(j)
    test = slim_roads[array]
    fig, ax = plt.subplots(figsize=(10, 10))
    fig.suptitle(str(pd.to_datetime(array_item[i]).date()), fontsize=20)
    slim_roads.plot(color='gray',ax=ax)
    beats.plot(ax=ax)
    test.plot(color='red',ax=ax)
    plt.tight_layout()
    plt.savefig('foo'+str(i)+'.png')
'''<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13898873711957824, 'fraction_numerical': 0.009825065899832255, 'mean_word_length': 7.588477366255144, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '259277', 'license': 'MIT', 'n_tokens_mistral': 1500, 'n_tokens_neox': 1427, 'n_words': 239, 'path': 'notebooks/dataset_work.ipynb', 'repo_name': 'nhtsai/datathon2019', 'size': '4173'}