<jupyter_start><jupyter_text>Exploratory Data Analysis for US Accidents 

 Kaggle description for the data set
 The US is a countrywide car accident dataset, which covers 49 states of the USA. The accident data are collected from February 2016 to Dec 2020, using two APIs that provide streaming traffic incident (or event) data. These APIs broadcast traffic data captured by a variety of entities, such as the US and state departments of transportation, law enforcement agencies, traffic cameras, and traffic sensors within the road-networks. Currently, there are about 4.2 million accident records in this dataset.



This dataset has been collected in real-time, using multiple Traffic APIs. Currently, it contains accident data that are collected from February 2016 to Dec 2020 for the Contiguous United States.



 Summary


*   Informaiton about accidents
*   Can bse useful to prevent accidents
  Installation of Python Libraries<jupyter_code>#Library for dataset downloading
!pip install opendatasets --upgrade --quiet<jupyter_output><empty_output><jupyter_text>Import data from Source<jupyter_code>import opendatasets as od

download_url = 'https://www.kaggle.com/sobhanmoosavi/us-accidents'

od.download(download_url)
# Import data set from CSV
data_filename = './us-accidents/US_Accidents_Dec20.csv'<jupyter_output><empty_output><jupyter_text>Data Preparation and Cleaning
* Load the file using Pandas
* Look at some information about the data & the columns
* Fix any missing or incorrect values<jupyter_code>import pandas as pd

df = pd.read_csv(data_filename)
df<jupyter_output><empty_output><jupyter_text>Summary of Data<jupyter_code>df.info()
df.describe()
# Find the numeric columns in the Ddata set
numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']

numeric_df = df.select_dtypes(include=numerics)
len(numeric_df.columns)<jupyter_output><empty_output><jupyter_text>Missing Value Count and Percentage<jupyter_code>df.isna().sum().sort_values(ascending=False)
# Percentages of missing values by column
missing_percentages = df.isna().sum().sort_values(ascending=False) / len(df)
missing_percentages*100 #%
type(missing_percentages)<jupyter_output><empty_output><jupyter_text>Missing Values Visualization<jupyter_code>#missing_percentages[missing_percentages > 0].plot(kind='bar')
import numpy as np
import matplotlib.pyplot as plt

plt.style.use('fivethirtyeight')
missing_df = df.isnull().sum(axis=0).reset_index()
missing_df.columns = ['column_name','missing_count']
missing_df = missing_df.loc[missing_df['missing_count']>0]
missing_df = missing_df.sort_values(by='missing_count')

ind = np.arange(missing_df.shape[0])
width = 0.5
fig,ax = plt.subplots(figsize=(12,10))
rects = ax.barh(ind,missing_df.missing_count.values,color='green')
ax.set_yticks(ind)
ax.set_yticklabels(missing_df.column_name.values, rotation='horizontal')
ax.set_xlabel("Count of missing values")
ax.set_title("Number of missing values in each column")
plt.show()<jupyter_output><empty_output><jupyter_text>Exploratory Analysis and Visualization
Columns wil be analyzed:

* City / States
* Start Lat, Start Lng
* Start Time
* Temperature
* Weather Condition  Visualization by City
Get the number of unique cities in the Data set<jupyter_code>cities = df.City.unique()
len(cities)<jupyter_output><empty_output><jupyter_text>Numer of accidents by City<jupyter_code>cities_by_accident = df.City.value_counts()
cities_by_accident.head(20)
cities_by_accident[:20].sort_values(ascending=True).plot(kind='barh')
fig,ax=plt.subplots(1,2,figsize=(16,6))

#Bar Plotting
df.City.value_counts().sort_values(ascending=False)[:20].sort_values().plot(kind='barh',ax=ax[0])
ax[0].set_title("Top 20 Accident Prone Cities",size=30)
ax[0].set_xlabel('Cities',size=14)

# Pie Chart Plotting
count=df['City'].value_counts()
groups=list(df['City'].value_counts().index)[:20]
counts=list(count[:20])
counts.append(count.agg(sum)-count[:20].agg('sum'))
groups.append('Other')
type_dict=pd.DataFrame({"group":groups,"counts":counts})
clr1=('brown','darksalmon','orange','hotpink','cadetblue','purple','red','gold','forestgreen','blue','plum')
qx = type_dict.plot(kind='pie', y='counts', labels=groups,colors=clr1,autopct='%1.1f%%', pctdistance=0.9, radius=1.2,ax=ax[1])
plt.legend(loc=0, bbox_to_anchor=(-0.01,1.1)) 
plt.subplots_adjust(wspace =0.5, hspace =0)
plt.ioff()
plt.ylabel('')
pass<jupyter_output><empty_output><jupyter_text>Visualization by State
The States with more accidents are CA, TX and FL<jupyter_code>states_accident = df.State.value_counts()
states_accident.head(20)
fig,ax=plt.subplots(1,2,figsize=(12,6))

#Bar Plotting
clr = ("blue", "forestgreen", "gold", "red", "purple",'cadetblue','hotpink','orange','darksalmon','brown')
df.State.value_counts().sort_values(ascending=False)[:10].sort_values().plot(kind='barh',color=clr,ax=ax[0])
ax[0].set_title("Top 10 Accident Prone States",size=20)
ax[0].set_xlabel('States',size=18)

# Pie Chart Plotting
count=df['State'].value_counts()
groups=list(df['State'].value_counts().index)[:10]
counts=list(count[:10])
counts.append(count.agg(sum)-count[:10].agg('sum'))
groups.append('Other')
type_dict=pd.DataFrame({"group":groups,"counts":counts})
clr1=('brown','darksalmon','orange','hotpink','cadetblue','purple','red','gold','forestgreen','blue','plum')
qx = type_dict.plot(kind='pie', y='counts', labels=groups,colors=clr1,autopct='%1.1f%%', pctdistance=0.9, radius=1.2,ax=ax[1])
plt.legend(loc=0, bbox_to_anchor=(1.15,0.4)) 
plt.subplots_adjust(wspace =0.5, hspace =0)
plt.ioff()
plt.ylabel('')
pass<jupyter_output><empty_output><jupyter_text>Visualization by Start Latitude & Longitude<jupyter_code>import seaborn as sns
sns.set_style("darkgrid")

df.Start_Lat
df.Start_Lng
sample_df = df.sample(int(0.1 * len(df)))
sns.scatterplot(x=sample_df.Start_Lng, y=sample_df.Start_Lat, size=0.001)
import folium
from folium.plugins import HeatMap

lat, lon = df.Start_Lat[0], df.Start_Lng[0]

for x in df[['Start_Lat', 'Start_Lng']].sample(100).iteritems():
    print(x[1])
zip(list(df.Start_Lat), list(df.Start_Lng))
sample_df = df.sample(int(0.001 * len(df)))
lat_lon_pairs = list(zip(list(sample_df.Start_Lat), list(sample_df.Start_Lng)))
map = folium.Map()
HeatMap(lat_lon_pairs).add_to(map)
map<jupyter_output><empty_output><jupyter_text>Visualization by Start Time<jupyter_code>import numpy as np
df['Start_Time'].head(5)
df.Start_Time = pd.to_datetime(df.Start_Time)
df['Start_Time'].head(5)<jupyter_output><empty_output><jupyter_text>**Hour Insights**
* A high percentage of accidents occur between 6 am to 10 am 
*Next higest percentage is 3 pm to 6 pm.<jupyter_code>import seaborn as sns

sns.set_style('whitegrid')
sns.set_context('talk')
sns.set_palette('GnBu_d')

sns.histplot(data=df.Start_Time.dt.hour, bins=24, kde=False)<jupyter_output><empty_output><jupyter_text>**Week days and Weekends Insights**
* Week days are more accidents than weekends<jupyter_code>sns.histplot(data=df.Start_Time.dt.dayofweek, bins=7, kde=False)<jupyter_output><empty_output><jupyter_text>**Month Insights**
* More accidents are in Winter<jupyter_code>sns.histplot(data=df.Start_Time.dt.month, bins=12, kde=False)<jupyter_output><empty_output><jupyter_text>Insights


* Less than 5% of cities have more than 1000 yearly accidents.
* Over 1200 cities have reported just one accident (need to investigate)<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13934535738142953, 'fraction_numerical': 0.022712090848363394, 'mean_word_length': 7.373601789709173, 'pattern_counts': {'":': 4, '<': 61, '<?xml version=': 0, '>': 63, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '116753', 'license': 'MIT', 'n_tokens_mistral': 2820, 'n_tokens_neox': 2683, 'n_words': 525, 'path': 'EDA_US_Accidents.ipynb', 'repo_name': 'maegop/Exploratory-Data-Analysis-for-US-Accidents', 'size': '7485'}