<jupyter_start><jupyter_text>Exploratory Data Analysis of Road Accidents in USA- toc: true- badges: true  IntroductionEvery year 1.3 million people die as a result of a road traffic crash around the world. And 20 - 50 million people suffer non-fatal injuries, with many incurring a disability as a result of their injury.[1 ](https://www.who.int/news-room/fact-sheets/detail/road-traffic-injuries) In USA alone, there were 33,244 fatal motor vehicle crashes in 2019 in which 36,096 deaths occurred.[2 ](https://www.iihs.org/topics/fatality-statistics/detail/state-by-state:~:text=There%20were%2033%2C244%20fatal%20motor%20vehicle%20crashes%20in%20the%20United%20States%20in%202019%20in%20which%2036%2C096%20deaths%20occurred.%20This%20resulted%20in%2011.0%20deaths%20per%20100%2C000%20people%20and%201.11%20deaths%20per%20100%20million%20miles%20traveled.) In this project, we are going to study the ['US Accidents'](https://www.kaggle.com/sobhanmoosavi/us-accidents) dataset provided by [<NAME>](https://www.kaggle.com/sobhanmoosavi) on https://www.kaggle.com. We can analyse this data to gain some really interesting insights about road accidents in USA, such as the impact of environmental stimuli on accidental occurance, the change in the occurance of accidents with change in months, or which 5 states have the highest (or lowest) number of accidents. [Note: This Dataset covers 49 mainland states of the USA (excluding Alaska) for the time period: February 2016 to Dec 2020]We will use the approach of Exploratory Data Analysis (EDA) for studying this data, which is used to analyse datasets to summarize their main characteristics, often using statistical graphics and other data visualization methods. EDA can help in seeing what the data can tell us beyond the formal modeling or hypothesis testing task.[3 ](https://en.wikipedia.org/wiki/Exploratory_data_analysis:~:text=In%20statistics%2C%20exploratory,hypothesis%20testing%20task) Because this dataset is huge, with dozens of features, it can be used to answer a lot of questions. For this project, we are going to limit ourselves to 5-10 questions and then try to answer these questions.  SetupImport the necessary libraries and get the file_path for the dataset.<jupyter_code>import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import matplotlib.pyplot as plt # data visualization
import seaborn as sns # data visualization

import os
for dirname, _, filenames in os.walk('US_Accidents_Dec20_updated.csv/'):
    for filename in filenames:
        print(os.path.join(dirname, filename))

# Read the file
file_path = "US_Accidents_Dec20_updated.csv/US_Accidents_Dec20_updated.csv"
us_accidents = pd.read_csv(file_path)
pd.set_option("display.max_columns", None)
us_accidents.head()<jupyter_output><empty_output><jupyter_text>Data Preparation and Cleaning  Premilinary analysis of dataset<jupyter_code>us_accidents.info()

# basic statistics
us_accidents.describe()<jupyter_output><empty_output><jupyter_text>Number of null values in the dataset, column vise<jupyter_code>null_val_cols = us_accidents.isnull().sum()
null_val_cols = null_val_cols[null_val_cols > 0].sort_values(ascending = False)
null_val_cols

# plot percentage of missing values
missing_percentage = null_val_cols / len(us_accidents) * 100
missing_percentage.plot(kind = 'barh', figsize = (14, 8))
plt.xlabel("Percentage of missing values")
plt.ylabel("Columns")
plt.show()<jupyter_output><empty_output><jupyter_text>Observations- The `Number` column represents the street number in address record. - It has over a million null values. Because we already have the location data from the `Start_Lat`, `Start_Lng`, `County` and `State` features, we can choose to drop this column from the dataset. With regards to columns representing weather conditions, we can choose to drop the columns having a significant number of null values, and impute other columns to the mean values from the same State and month, which should represent similar weather conditions.<jupyter_code>us_accidents.drop(labels = ["Number", "Precipitation(in)"], axis = 1, inplace = True)
us_accidents.shape<jupyter_output><empty_output><jupyter_text>We'll also convert the `Start_Time` column to datetime dtype, so that we can use use it to apply datetime functions later on.<jupyter_code>us_accidents.Start_Time = us_accidents.Start_Time.astype("datetime64")<jupyter_output><empty_output><jupyter_text>Exploratory Analysis and Visualization**With these columns in mind, we will try to answer the following questions from the dataset:**1. Are there more accidents in warmer or colder areas?2. Which 5 states have the highest number of accidents?3. Among the top 100 cities in number of accidents, which states do they belong to most frequently.4. What time of the day are accidents most frequent in?5. Which days of the week have the most accidents?6. Which months have the most accidents?7. What is the trend of accidents year over year (decreasing/increasing?)8. How does accident severity change with change in precipitation? We can start with plotting longitude and lattitude data to get a sense of where is the data concentrated<jupyter_code>lat, long = us_accidents.Start_Lat, us_accidents.Start_Lng

# plot locations of accidents
plt.figure(figsize = (10, 6))
sns.scatterplot(x = long, y = lat, s = 0.5)<jupyter_output><empty_output><jupyter_text>It appears that accidents are more concentrated near the coasts, which are more populated areas of US. There seems to be a sharp decline in the mid - mid-western US. Apart from lack of accidents in those areas, it could also be a signal of poor data collection in those states.   Q1. Are there more accidents in warmer or colder areas? First, we'll plot a histogram for the temperature data.<jupyter_code>us_accidents["Temperature(F)"].plot(kind = "hist", logy = True)
plt.xlabel("Temperature(F)")
plt.show()
print(f'''Mean: {us_accidents["Temperature(F)"].mean()} 
Skewness: {us_accidents["Temperature(F)"].skew()}
Kurtosis: {us_accidents["Temperature(F)"].kurtosis()}''')<jupyter_output>Mean: 58.5816432896377 
Skewness: -0.30533266112446805
Kurtosis: -0.13261978426888454<jupyter_text>Observations- Data is normally distributed with very low skewness and kurtosis values. - Thus, the data doesn't support the idea that warmer areas have more accidents than colder ones or vice versa.   Q2. Which 5 states have the highest number of accidents?<jupyter_code>accidents_by_states = us_accidents.State.value_counts()
print("The top 5 states in terms of number of accidents are:")
accidents_by_states.head()
plt.figure(figsize = (16, 5))
sns.barplot(x = accidents_by_states.index, y = accidents_by_states.values)<jupyter_output><empty_output><jupyter_text>Q3. Among the top 100 cities in number of accidents, which states do they belong to most frequently.<jupyter_code>city_accidents = us_accidents.City.value_counts()
city_accidents
city_states = us_accidents.groupby("City")["State"].aggregate(pd.Series.mode)
city_states
top_100_cities = pd.concat([city_accidents, city_states], axis = 1)[:100]
top_100_cities

#Most frequent states in the top 100 cities in the number of accidents
most_freq_states = top_100_cities.State.value_counts()
most_freq_states

# plot
most_freq_states.plot(kind = "bar", figsize = (15,5))
plt.xlabel("States")
plt.ylabel("Frequency in top 100 cities by number of accidents")
plt.title("Most frequent states in the top 100 cities in the number of accidents")
plt.show()<jupyter_output><empty_output><jupyter_text>Q4. What time of the day are accidents most frequent in? Do more accidents tend to occur at a particular time of the day?<jupyter_code># plot
us_accidents.Start_Time.dt.hour.plot(kind = 'hist', bins = 24)
plt.xlabel("Hour of the day")
plt.show()<jupyter_output><empty_output><jupyter_text>ObservationsIt appears that accidents tend to occur more in the morning between 7-9 and in the afternoon between 13-17. Office hours could be the reason behind this trend. We can examine this by separating the data for weekdays and weekends.<jupyter_code># plot
weekdays = us_accidents.Start_Time.dt.dayofweek < 5
us_accidents.loc[weekdays].Start_Time.dt.hour.plot(kind = 'hist', bins = 24)
plt.xlabel("Hour of the day")
plt.title("Accidents frequency of Weekdays")
plt.show()

# plot data for weekends
us_accidents.loc[~weekdays].Start_Time.dt.hour.plot(kind = 'hist', bins = 24)
plt.xlabel("Hour of the day")
plt.title("Accidents frequency of Weekends")
plt.show()<jupyter_output><empty_output><jupyter_text>Observations- On weekends, there seems to be a breakaway from the pattern that we saw earlier, which supports the idea that the trend we saw earlier could be due to the traffic resulting from office timings, which are usually closed on weekends. - On weekends, accidents tend to occur more during daylight, which should be due to more traffic during those hours.  Q5. Which days of the week have the most accidents?<jupyter_code>sns.histplot(us_accidents.Start_Time.dt.dayofweek, bins = 7)
plt.xlabel("Day of Week")
plt.show()<jupyter_output><empty_output><jupyter_text>Observations- Accidents occur more on weekdays and there is a sharp drop in their count on the weekends.   Q6. Which months have the most accidents?<jupyter_code>sns.histplot(us_accidents.Start_Time.dt.month, bins = 12)
plt.xlabel("Month")
plt.show()<jupyter_output><empty_output><jupyter_text>Observations- Later months of the year appear to witness more accidents. This cannot be due to temperature(winter season) because the count drops suddenly in the starting months of the year. Due to there being no other apparent cause, it demands further analysis.We can start by looking at the trend year over year.<jupyter_code>fig, axes = plt.subplots(2, 3, figsize = (15,10), )
year = 2016
for r in range(2):
    for c in range(3):
        if year < 2021:
            year_data = us_accidents.loc[us_accidents.Start_Time.dt.year == year]
            sns.histplot(year_data.Start_Time.dt.month, bins = 12, ax = axes[r, c])
            axes[r, c].set_title(year)
            axes[r, c].set_xlabel("Month of the year")
            year += 1<jupyter_output><empty_output><jupyter_text>Observations- Probably because the data was started being collected in the year 2016, the starting months have a much lower number of datapoints. - Also, in the year 2020, because of the coronavirus pandemic lockdown restrictions, there was a suddent drop in the middle of the year. And as the restrictions eased, more accidents started to occur.But this data requires further more analysis on the month wise trends.   Q7. What is the trend of accidents year over year (decreasing/increasing?)<jupyter_code>us_accidents.Start_Time.dt.year.plot(kind = 'hist', bins = 5, title = "Year wise trend", 
                                     xticks = np.arange(2016, 2021), figsize = (7, 5))
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0860047627770654, 'fraction_numerical': 0.029492581058801978, 'mean_word_length': 5.557957957957958, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 62, 'https://': 6, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '424675', 'license': 'Apache-2.0', 'n_tokens_mistral': 3546, 'n_tokens_neox': 3230, 'n_words': 1201, 'path': '_notebooks/2021-06-01-exploratory-data-analysis-of-road-accidents-in-usa.ipynb', 'repo_name': 'ncitshubham/blogs', 'size': '10926'}