<jupyter_start><jupyter_text>The ChallengeThe goal of this algorithm is to interpret historical accident data and predict the severity of accidents that may occur in the area based on the attributes that are heavily correlated. This information can be used to suggest avoiding areas during route planning and could allow the insurance company to make preliminary estimates based on the accident location  Data VisualizationReads in accident data then processes it to produce a bar graph measuring accidents by state. This would allow the analyzer to visualize the data to see the amount of accidents that occur in each state, compared to other states.<jupyter_code>path = os.getcwd()
data_df = pd.read_csv((path+'/Static/US_Accidents_Dec19.csv'),index_col=False)
data_df = data_df.drop(columns = ['Start_Lat','Start_Lng','End_Lat','End_Lng','Source','TMC'])
#df = data_df.drop(['ID'],axis = 1)
data_df.head(5)
[row,col] = data_df.shape
state_count = ((data_df.groupby('State').count())['ID'])
plt.figure(figsize=(15,10))
ax = sns.barplot(state_count.index, state_count.values)
plt.ylabel('Number of accidents')<jupyter_output><empty_output><jupyter_text>California and Texas are states with the most number of accidents. Since Texas and California are the most populous states, because of the number of people drives in these states is also the highest.<jupyter_code>#df = data_df.drop(['ID'],axis = 1)
data_df.head(5)
[row,col] = data_df.shape
state_count = ((data_df.groupby('State').count())['ID'])
plt.figure(figsize=(15,10))
ax = sns.barplot(state_count.index, state_count.values/row)
plt.ylabel('Percentage of accidents')<jupyter_output><empty_output><jupyter_text>Over 20% of all accident in the United States occurs in the state of California, and over 10% of accidents occur in Texas.<jupyter_code>state_population_df = pd.read_csv((path+'/Static/state_population_copy.csv'),index_col= 'Postal')
plt.figure(figsize=(15,10))
state_population = state_population_df.drop(index = ['AR','IA'])
state_population_df = state_population_df.sort_values(by = 'Postal')
state_population_df['count'] = state_count
plt.figure(figsize=(15,10))
state_population_df = state_population_df.dropna()
state_population_df.head()
ax = sns.barplot(state_count.index, 
                 100*state_population_df['count']/state_population_df['Population'])
plt.ylabel('Ratio of accident to number of people')
old_series = state_population_df['count']/state_population_df['Population']
new_series = pd.Series(state_count.values/row, index = state_count.index )
new_series = new_series.drop(index = ['AR','IA'])
new_df = pd.concat([new_series,old_series],axis = 1,sort= True)
new_df.columns = ['percentage','average']
new_df = new_df.dropna()
#new_df.to_csv('Number of accidents.csv')<jupyter_output><empty_output><jupyter_text>To get a better understanding of the frequency of accidents relative to population (weight), we divide the number of accidents by population. This shows that South Dakota has the worst number of accidents compared to their population.<jupyter_code>plt.figure(figsize=(15,10))
test = (pd.to_datetime(data_df['End_Time']) - pd.to_datetime(data_df['Start_Time']))/np.timedelta64(1,'m')
sns.scatterplot(x = data_df.index,y = test , hue = data_df['Severity'])
plt.xlabel('Id')
plt.ylabel('Duration')
plt.title('Accident times')<jupyter_output><empty_output><jupyter_text>The figure displays the time during each accident in the data set.  As one can see, the majority of accident duration is less than 100 minutes and of low severity.   Pre-proccessingIn order to further understand which attributes effect the severity of an accident the most, we process the data to find the attributes with the strongest correlation. This leaves us with fators that actually play a part in accident severity.<jupyter_code>cols = ('Street','Side', 'City', 'County', 'State', 'Zipcode', 'Country', 'Timezone',
       'Airport_Code', 'Weather_Timestamp','Sunrise_Sunset', 'Civil_Twilight', 'Nautical_Twilight',
       'Astronomical_Twilight')
new_data_df = data_df.drop(columns = ['End_Time','Start_Time'])
for c in cols:
    lbl = LabelEncoder() 
    lbl.fit(list(new_data_df[c].values)) 
    new_data_df[c] = lbl.transform(list(new_data_df[c].values))
incode = LabelEncoder()
new_data_df['Weather_Condition'] = incode.fit_transform(list(new_data_df['Weather_Condition'].values))<jupyter_output><empty_output><jupyter_text>To determine the which attributes effect severity, we encode all categorical attributes and calculate the corralation rates.<jupyter_code>coor_value = ((new_data_df.corr()['Severity']).abs())
coor_value.sort_values()
X_data = data_df[(['Crossing','Side', 'Wind_Chill(F)','Astronomical_Twilight','Traffic_Signal','Severity']) ]
X_data = X_data.dropna()
y_label = X_data['Severity']
X_data = X_data.drop(columns = 'Severity',axis = 1)<jupyter_output><empty_output><jupyter_text>Feature selectionUsing the information gained from the the correlation, between the different features and sevarity of the accident.<jupyter_code>cols = ('Crossing','Side','Astronomical_Twilight','Traffic_Signal')
for c in cols:
    lbl = LabelEncoder() 
    lbl.fit(list(X_data[c].values)) 
    X_data[c] = lbl.transform(list(X_data[c].values))
X_data = X_data.dropna()
X_train, X_Valid, Y_train, Y_Valid=train_test_split(X_data,y_label, test_size=0.15, random_state=4)<jupyter_output><empty_output><jupyter_text>Model ExplorationCreate training and actual sets by splitting the given data to create the model that will be used to predict severity. Assessing the perfromance of different model, and hyperparameters.<jupyter_code>model_rf = RandomForestClassifier(n_estimators = 100)
model_rf.fit(X_train,Y_train)
prediction = model_rf.predict(X_Valid)
rf_accuracy = metrics.accuracy_score(prediction, Y_Valid)
print('Random_Forest acc:', rf_accuracy)<jupyter_output>Random_Forest acc: 0.6985789257843842<jupyter_text>We assess the performance of the AdaBoost Classifier and restrict the number of estimators to increase training time.<jupyter_code>model_ada = AdaBoostClassifier(n_estimators= 100)
model_ada.fit(X_train,Y_train)
prediction = model_ada.predict(X_Valid)
ada_accuracy = metrics.accuracy_score(prediction, Y_Valid)
print('Random_Forest acc:', ada_accuracy)<jupyter_output>Random_Forest acc: 0.6982936411237838<jupyter_text>Train the two model on all of the data, and saving the model to be used in the web application for determining the safety of a route.<jupyter_code># Random Forest Model
model_rf = RandomForestClassifier(n_estimators = 100)
model_rf.fit(X_data,y_label)
pkl_filename = "random_forest.pkl"
with open(pkl_filename, 'wb') as file:
    pickle.dump(model_rf, file)
# The AdaBoost Model
model_ada = AdaBoostClassifier(n_estimators= 100)
model_ada.fit(X_data,y_label)
pkl_filename = "ada_boost.pkl"
with open(pkl_filename, 'wb') as file:
    pickle.dump(model_rf, file)<jupyter_output><empty_output><jupyter_text>Creating a final CSV which will be uploaded to google's autoML platform, and used to train a neural network.<jupyter_code>final_data = X_train
final_data['label'] = y_label
final_data= final_data.reset_index(drop = True)
final_data.head()
final_data.to_csv((path+'Final_data.csv'))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11192483073096587, 'fraction_numerical': 0.012574271106812215, 'mean_word_length': 7.377314814814815, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '351877', 'license': 'MIT', 'n_tokens_mistral': 2456, 'n_tokens_neox': 2321, 'n_words': 600, 'path': 'main.ipynb', 'repo_name': 'Avashist1998/TamuMake2020_Webapp', 'size': '7237'}