<jupyter_start><jupyter_text>Introduction/Business Problem Car accidents have a huge social, economic and environmental impact to our world. Approximately 54 million people sustained injuries from car accidents each year and more than 1.4 millions resulted to deaths. Road traffic injuries are the leading cause of death for children and young adults aged 5-29 years. The road traffic crashes costs more than 3% of GDP.It would be hugly important to predict the severity of a car accident and warn the drivers,in advance and on time, in order to drive more savely or even to adjust their journey.Thus, the aim of this project is to apply quantitative methods and create a model that predicts the severity of a car accident based on weather and road conditions, traffic jam etc.The dataset are from Kaggle and contains details of traffic accidents in pennsylvania State of United States for the years of 2016 - 2020  Let's import our Data and Libraries<jupyter_code>import pandas as pd
df = pd.read_csv('last.csv')
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
pd.set_option('display.max_rows', 200)
pd.set_option('display.max_columns', 200)
plt.style.use('ggplot')

from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.model_selection import GridSearchCV
from sklearn.feature_selection import SelectFromModel
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
from sklearn.metrics import roc_curve, auc<jupyter_output><empty_output><jupyter_text>Let's have a quick look at our data<jupyter_code>df.head(3)<jupyter_output><empty_output><jupyter_text>Edit date and time columns<jupyter_code>df['Start_Time'] = pd.to_datetime(df['Start_Time'], errors='coerce')
df['End_Time'] = pd.to_datetime(df['End_Time'], errors='coerce')

df['Year']=df['Start_Time'].dt.year
df['Month']=df['Start_Time'].dt.strftime('%B')
df['Day']=df['Start_Time'].dt.day
df['Hour']=df['Start_Time'].dt.hour
df['Weekday']=df['Start_Time'].dt.strftime('%a')

td='Time_Duration(min)'
df[td]=round((df['End_Time']-df['Start_Time'])/np.timedelta64(1,'m'))<jupyter_output><empty_output><jupyter_text>Visualize some key statistics<jupyter_code>### Plot some boolean values
fig = plt.figure(figsize=(10,10)) 
fig_dims = (3, 2)

plt.subplot2grid(fig_dims, (0, 0))
df['Amenity'].value_counts().plot(kind='bar', color ='green',
                                     title='Amenity')
plt.subplot2grid(fig_dims, (0, 1))
df['Crossing'].value_counts().plot(kind='bar',color = 'blue',
                                     title='Crossing')
plt.subplot2grid(fig_dims, (1, 0))
df['Junction'].value_counts().plot(kind='bar',color = 'yellow',
                                     title='Junction')
plt.subplot2grid(fig_dims, (1, 1))
df['Roundabout'].value_counts().plot(kind='bar',color = 'red',
                                     title='Roundabout')        
plt.subplot2grid(fig_dims, (2, 0))
df['Stop'].value_counts().plot(kind='bar',color = 'black',
                                     title='Stop') 
plt.subplot2grid(fig_dims, (2, 1))
df['Station'].value_counts().plot(kind='bar',color = 'pink',
                                     title='Station') 
### The severity distribution
f,ax=plt.subplots(1,2,figsize=(12,4))
df['Severity'].value_counts().plot.pie(autopct='%1.1f%%',ax=ax[0],shadow=True)
ax[0].set_title('Percentage Severity Distribution')
ax[0].set_ylabel('Number')
sns.countplot('Severity',data=df,ax=ax[1],order=df['Severity'].value_counts().index)
ax[1].set_title('Count of Severity')
plt.show()
### Severity based on Temperature

bins = np.linspace(df['Temperature(F)'].min(), df['Temperature(F)'].max(), 10)
g = sns.FacetGrid(df, col='Severity', palette="Set2", col_wrap=4)
g.map(plt.hist, 'Temperature(F)', bins=bins, ec="k")

plt.show()
fig = plt.figure(figsize=(10,10)) 
fig_dims = (3, 2)

plt.subplot2grid(fig_dims, (0, 0))
df['Sunrise_Sunset'].value_counts().plot(kind='bar', color ='green',
                                     title='Sunrise_Sunset')
plt.subplot2grid(fig_dims, (0, 1))
df['Nautical_Twilight'].value_counts().plot(kind='bar',color = 'blue',
                                     title='Nautical_Twilight ')
plt.subplot2grid(fig_dims, (1, 0))
df['Civil_Twilight'].value_counts().plot(kind='bar',color = 'yellow',
                                     title='Civil_Twilight ')
plt.subplot2grid(fig_dims, (1, 1))
df['Astronomical_Twilight'].value_counts().plot(kind='bar',color = 'red',
                                     title='Astronomical_Twilight ')
df.groupby("Month").size().plot(kind="line",color = 'green', title= 'By Month')
df.groupby("Weekday").size().plot(kind="bar",color = 'blue', title = 'By Weekday')
### Top 10 cities for accidents 

fig, ax=plt.subplots(figsize=(10,6))
df['City'].value_counts().sort_values(ascending=False).head(10).plot.bar(width=0.5,edgecolor='k',align='center',linewidth=2)
plt.xlabel('City',fontsize=2)
plt.ylabel('Number',fontsize=2)
ax.tick_params(labelsize=20)
plt.title('Top 10 Cities for accidents',fontsize=12)
plt.grid()
plt.ioff()<jupyter_output><empty_output><jupyter_text>Clean some rows and columns<jupyter_code># Remove outliers and negative values
df[td][df[td]<=0]
neg_outliers=df[td]<=0
df[neg_outliers] = np.nan

df.dropna(subset=[td],axis=0,inplace=True)

n=3
median = df[td].median()
std = df[td].std()
outliers = (df[td] - median).abs() > std*n
df[outliers] = np.nan
df[td].fillna(median, inplace=True)

df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 106787 entries, 0 to 106786
Data columns (total 56 columns):
 #   Column                 Non-Null Count   Dtype         
---  ------                 --------------   -----         
 0   Unnamed: 0             106766 non-null  float64       
 1   ID                     106766 non-null  object        
 2   Source                 106766 non-null  object        
 3   TMC                    86366 non-null   float64       
 4   Severity               106766 non-null  float64       
 5   Start_Time             106766 non-null  datetime64[ns]
 6   End_Time               106766 non-null  datetime64[ns]
 7   Start_Lat              106766 non-null  float64       
 8   Start_Lng              106766 non-null  float64       
 9   End_Lat                20400 non-null   float64       
 10  End_Lng                20400 non-null   float64       
 11  Distance(mi)           106766 non-null  float64       
 12  Description            106766 non-null  obje[...]<jupyter_text>Drop some columns<jupyter_code>feature_lst=['Source','TMC','Severity','Start_Lng','Start_Lat','Distance(mi)','Side','City','County','State','Timezone','Temperature(F)','Humidity(%)','Pressure(in)', 'Visibility(mi)','Weather_Condition','Amenity','Bump','Crossing','Give_Way','Junction','No_Exit','Railway','Roundabout','Station','Stop','Traffic_Calming','Traffic_Signal','Turning_Loop','Sunrise_Sunset','Hour','Weekday', 'Time_Duration(min)']
df=df[feature_lst].copy()
df.info()
df.dropna(subset=df.columns[df.isnull().mean()!=0], how='any', axis=0, inplace=True)
df.shape<jupyter_output><empty_output><jupyter_text>Transform the categorigal values<jupyter_code>df_dummy = pd.get_dummies(df,drop_first=True)
df_dummy.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 84573 entries, 0 to 86365
Columns: 1219 entries, TMC to Weekday_Wed
dtypes: float64(24), uint8(1195)
memory usage: 112.5 MB<jupyter_text>Predict the severity of an accident by different Machine Learning methods<jupyter_code># Create training and testing data 

df=df_dummy
target='Severity'

y = df[target]
X = df.drop(target, axis=1)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=21, stratify=y)
# Algorithms that will be used 
algo_lst=['Logistic Regression','Decision Trees',' K-Nearest Neighbors']
accuracy_lst=[]<jupyter_output><empty_output><jupyter_text>Logistic regression<jupyter_code>lr = LogisticRegression(random_state=0)
lr.fit(X_train,y_train)
y_pred=lr.predict(X_test)

acc=accuracy_score(y_test, y_pred)
accuracy_lst.append(acc)

print('The accuracy_score of logistics regression is: {:.3f}.'.format(acc))<jupyter_output>/home/jupyterlab/conda/envs/python/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:433: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
  FutureWarning)
/home/jupyterlab/conda/envs/python/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:460: FutureWarning: Default multi_class will be changed to 'auto' in 0.22. Specify the multi_class option to silence this warning.
  "this warning.", FutureWarning)<jupyter_text>Decision tree<jupyter_code>dt_entropy = DecisionTreeClassifier(max_depth=8, criterion='entropy', random_state=1)
dt_entropy.fit(X_train, y_train)
y_pred= dt_entropy.predict(X_test)

accuracy_entropy = accuracy_score(y_test, y_pred)

print('The accuracy_score of Decision Tree entropy is: {:.3f}.'.format(accuracy_entropy))

dt_gini = DecisionTreeClassifier(max_depth=8, criterion='gini', random_state=1)
dt_gini.fit(X_train, y_train)
y_pred= dt_gini.predict(X_test)

accuracy_gini = accuracy_score(y_test, y_pred)

acc=accuracy_gini
accuracy_lst.append(acc)

print('The accuracy_score of Decision Tree gini is: {:.3f}.'.format(accuracy_gini))<jupyter_output>The accuracy_score of Decision Tree entropy is: 0.869.
The accuracy_score of Decision Tree gini is: 0.874.<jupyter_text>K-Nearest Neighbors<jupyter_code>knn = KNeighborsClassifier(n_neighbors=6)
knn.fit(X_train,y_train)
y_pred = knn.predict(X_test)

acc=accuracy_score(y_test, y_pred)
accuracy_lst.append(acc)

print('[K-Nearest Neighbors (KNN)] knn.score: {:.3f}.'.format(knn.score(X_test, y_test)))
print('[K-Nearest Neighbors (KNN)] accuracy_score: {:.3f}.'.format(acc))<jupyter_output>[K-Nearest Neighbors (KNN)] knn.score: 0.812.
[K-Nearest Neighbors (KNN)] accuracy_score: 0.812.
{'fraction_non_alphanumeric': 0.1409203364670955, 'fraction_numerical': 0.031172686788718457, 'mean_word_length': 4.614444444444445, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '606324', 'license': 'MIT', 'n_tokens_mistral': 3724, 'n_tokens_neox': 3446, 'n_words': 704, 'path': 'Applied Data Science Capstone/Week 3/FINAL project.ipynb', 'repo_name': 'prashantwitty/IBM-Data-Science-Professional-Certificate', 'size': '10105'}