<jupyter_start><jupyter_text>Capstone Project - Predicting Accident Severity  Applied Data Science Capstone by IBM/Coursera  INTRODUCTION In this Project,we try to predict the severity of accidents taking into consideration many facts such as location,collision type,no. of vehicles involved etc.We all must have undertaken journeys at critical moments of time but in some cases there might be a dilemma about whether to pursue our journey or not. Sometimes we go on nevertheless and it may lead to fatal consequences. But we all can agree upon the fact that undertaking a journey on a cold rainy night with a possible chances of hurricanes are absolutely dangerous when compared to normal days. These are two extreme instances and we can choose our options wisely in these type of scenarios. But sometimes weather or some other factors may be change all of a sudden, and we may be a little too late to make any decisions in such a scenario.But what if some computer generated model can warn us about potential hazards ? Since data over the course of many years is available to computers , we can count on the predictions of the model since it will be at least as accurate as ours, provided the data is accurate and updated.  DATA Data I have chosen is the given csv file itself which contains about 37 columns like objectid,location,severity , person count etc. The given dataset is not complete i.e some missing values in some columns namely (x,y) , EXCEPTRSNDESC ,to name a fewThere are many columns with categorical values as well as some columns are purely numeric . Columns with categorical values will not help in regression estimation and the dataset should be trimmed adeqautely in order to obtain the correct sample for undergoing regression,The column Severity-Code contains two discrete values 1&2 which gives us information whether the accident is more/less severe. where 1 is less severe.There are of course many more columns which gives us important details such as weather , road condition,light condition etc which may be transformed into dummy columns with binary values for the ease of processing.Other data preparation involves balancing the labelled data, transformation, filling missing data, and cleaning the dataset etc.After appropriate sampled data is obtained , we can chose an appropriate machine learning model to fit the data and predict the estimates.At last we must also ensure that sufficient level of accuracy is obtained as well ! no. of cols : 37no.of rows : 194673  Importing Libraries<jupyter_code>import pandas as pd
import numpy as np
from sklearn import preprocessing
import matplotlib.pyplot as plt 
plt.rc("font", size=14)
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
import seaborn as sns
sns.set(style="white")
sns.set(style="whitegrid", color_codes=True)
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Getting data<jupyter_code>!wget -O Data-Collisons.csv https://s3.us.cloud-object-storage.appdomain.cloud/cf-courses-data/CognitiveClass/DP0701EN/version-2/Data-Collisions.csv<jupyter_output>--2020-09-09 07:15:52--  https://s3.us.cloud-object-storage.appdomain.cloud/cf-courses-data/CognitiveClass/DP0701EN/version-2/Data-Collisions.csv
Resolving s3.us.cloud-object-storage.appdomain.cloud (s3.us.cloud-object-storage.appdomain.cloud)... 172.16.17.32
Connecting to s3.us.cloud-object-storage.appdomain.cloud (s3.us.cloud-object-storage.appdomain.cloud)|172.16.17.32|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 73917638 (70M) [text/csv]
Saving to: ‘Data-Collisons.csv’

100%[======================================>] 73,917,638  36.6MB/s   in 1.9s   

2020-09-09 07:15:54 (36.6 MB/s) - ‘Data-Collisons.csv’ saved [73917638/73917638]<jupyter_text>Reading data to dataframe df !<jupyter_code>df=pd.read_csv('Data-Collisons.csv')
df.head(5)
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 194673 entries, 0 to 194672
Data columns (total 38 columns):
SEVERITYCODE      194673 non-null int64
X                 189339 non-null float64
Y                 189339 non-null float64
OBJECTID          194673 non-null int64
INCKEY            194673 non-null int64
COLDETKEY         194673 non-null int64
REPORTNO          194673 non-null object
STATUS            194673 non-null object
ADDRTYPE          192747 non-null object
INTKEY            65070 non-null float64
LOCATION          191996 non-null object
EXCEPTRSNCODE     84811 non-null object
EXCEPTRSNDESC     5638 non-null object
SEVERITYCODE.1    194673 non-null int64
SEVERITYDESC      194673 non-null object
COLLISIONTYPE     189769 non-null object
PERSONCOUNT       194673 non-null int64
PEDCOUNT          194673 non-null int64
PEDCYLCOUNT       194673 non-null int64
VEHCOUNT          194673 non-null int64
INCDATE           194673 non-null object
INCDTTM           194673 non-null obje[...]<jupyter_text>Exploring using value_counts() function<jupyter_code>df['SEVERITYCODE'].value_counts()
df['X'].value_counts()
df['Y'].value_counts()
df['INCKEY'].value_counts()
df['COLDETKEY'].value_counts()
df['REPORTNO'].value_counts()
df['STATUS'].value_counts()<jupyter_output><empty_output><jupyter_text>By exploring first few rows , we see that Severity code corresponds to SEVERITYDESC i.e 1 means property damage only collision 2 means injury collison ! X & Y are location and some values are missing INCKEY and COLDETkey has similar values and we can use object id which is a more reliable and continuos index Reportno has 3 ambiguos values so we wont consider it while sampling Status has no relevant metadata info !<jupyter_code># NOw exploring even more columns with value_counts()
df['ADDRTYPE'].value_counts()
df['INTKEY'].value_counts()
df['COLLISIONTYPE'].value_counts()
df['PERSONCOUNT'].value_counts()
df['PEDCOUNT'].value_counts()
df['VEHCOUNT'].value_counts()
df['UNDERINFL'].value_counts()
df['WEATHER'].value_counts()
df['ROADCOND'].value_counts()
df['LIGHTCOND'].value_counts()
df['ST_COLCODE'].value_counts()
df['SEGLANEKEY'].value_counts()
df['CROSSWALKKEY'].value_counts()
df['SDOT_COLCODE'].value_counts()
df['INATTENTIONIND'].value_counts()
df['EXCEPTRSNCODE'].value_counts()<jupyter_output><empty_output><jupyter_text>Some insights understood by exploring all the columns   EXCEPTRSNDESC : "not enough information"		//REMOVE EVERY ROW WITH "NEI" value in this columnEXCEPTRSNCODE : NEI					object id - index - because it is continuosly numberedINCKEY(unique key for incident) & COLDETKEY (secondary key) :       // contains pretty much same values except for very few values.REPORT NO., STATUS - no info3 pairs of report number have same report number ADDRTYPE : intersection,block,alley etc 3 Types only !STATUS : matched / unmatched 				NO METADATA IS AVAILABLE INTKEY : intersection key (only for addrtype=intersection)			//LOCATION : text values of loc. //not neeededSEVERITY CODE_1 : same as SEVERITY CODE can be dropped 		 	//CAN BE DROPPEDSEVERITY_DESC : 1--> "property damage" , 2--> injury collssion       		//CAN BE DROPPEDCOLLISION TYPE :	Parked Car    47987			Angles        34674			Rear Ended    34090			Other         23703			Sideswipe     18609			Left Turn     13703			Pedestrian     6608			Cycles         5415			Right Turn     2956			Head On        2024				Name: COLLISIONTYPE, dtype: int64 PERSONCOUNT : Double The total number of people involved in the collisionPEDCOUNT : Double The number of pedestrians involved in the collision. This is entered by the state.PEDCYLCOUNT : Double The number of bicycles involved in the collision.This is entered by the state.VEHCOUNT : Double The number of vehicles involved in the collision.INJURIES : Double The number of total injuries in the collision. Thisis entered by the state.SERIOUSINJURIES : Double The number of serious injuries in the collision.This is entered by the state.FATALITIES : Double The number of fatalities in the collision. This is entered by the state		INCDATE , INCDTTM : date and time and timestamp of incidentUNDERINFL - contains 0/1 and true/false both kind of values are present !WEATHER,ROADCOND,LIGHTCOND : can all be consideredPEDROWNOTGRNT : //MOST OF THE VALUES ARE NULL !!  			REMOVE THIS COLUMNSDOTCOLNUM : No. given by SDOT , can be removed !			MAY REMOVESPEEDING : MOst of the columns have a null value !			CAN BE REMOVED ST_COLCODE : Some values have redundant entries ie 11 have two 			can be removed		 IT is because numbers are entered both as string and int , so covert everything to int and remove rows with invlaid ST_colcode ie (>84,75-80 ) etc which are not in metadata         ST_COLDESC : CORRECTLY MApped to st_colcode , same problem !SDOT_COLDESC AND ST_COLDESC have some differencesINATTENTIONIND : MOst values are null !!`					can be removedSEGLANEKEY : A key for the lane segment in which the collision occurred.		CAN BE REMOVED !!		Most values are zeroes 		CROSSWALKKEY : same !! most are zeroes						can be removed !HITPARKEDCAR : NO null values ! either YES OR NO<jupyter_code>data=df.copy()
data.head()<jupyter_output><empty_output><jupyter_text>Removing EXCEPTRSNCODE = "NEI"<jupyter_code>nei_index=data[data['EXCEPTRSNCODE']=="NEI"].index #THERE ARE INSUFFICIENT INFO IN THESE ROWS 
data.drop(nei_index,inplace=True)
data.shape<jupyter_output><empty_output><jupyter_text>REmoving columns which are of no use !<jupyter_code>df=data.copy()
df.drop(['SEVERITYCODE.1','PEDROWNOTGRNT','SPEEDING','SEGLANEKEY','CROSSWALKKEY','INATTENTIONIND','COLDETKEY','EXCEPTRSNCODE','EXCEPTRSNDESC','SDOT_COLCODE','SDOT_COLDESC','SDOTCOLNUM'],axis=1,inplace=True)
df.shape
data=df.copy()
data.shape
data.index<jupyter_output><empty_output><jupyter_text>WE are only selecting columns which are of our use - by disregarding columns such as Location which need to be removed but they cannot be used in the sample<jupyter_code>data=data[['SEVERITYCODE','X','Y','OBJECTID','ADDRTYPE','COLLISIONTYPE','PERSONCOUNT','PEDCOUNT','VEHCOUNT','UNDERINFL','WEATHER','ROADCOND','LIGHTCOND','ST_COLCODE','HITPARKEDCAR']]
data.head()
data.shape
data.describe(include='all')
### we are also dropping X,Y co ordinates for the location as well since some values are missing and difference between max y and min y is just 0.23
df['Y'].max()-df['Y'].min()
data.drop(['X','Y'],axis=1,inplace=True)
data.shape
### THERE are some columns with null values !
df['ADDRTYPE'].isnull().value_counts()<jupyter_output><empty_output><jupyter_text>I found out that some columns have some null values , beacuse no.of rows of null values is very very small compared to total no.of rows , we can remove those rows!<jupyter_code>addrtypenull=data[data['ADDRTYPE'].isnull()==True].index
data.drop(addrtypenull,inplace=True)
collisiontypenull=data[data['COLLISIONTYPE'].isnull()==True].index
data.drop(collisiontypenull,inplace=True)

underinflnull=data[data['UNDERINFL'].isnull()==True].index
data.drop(underinflnull,inplace=True)
data.shape

weathernull=data[data['WEATHER'].isnull()==True].index
data.drop(weathernull,inplace=True)
data.shape

#data.describe(include='all')
roadcondnull=data[data['ROADCOND'].isnull()==True].index
data.drop(roadcondnull,inplace=True)

lightcondnull=data[data['LIGHTCOND'].isnull()==True].index
data.drop(lightcondnull,inplace=True)
data.describe(include='all')

data.shape

### Some 4546 rows are removed<jupyter_output><empty_output><jupyter_text>Data Transformation !<jupyter_code>#st_colcode column have both int and string values , so they need to be converted to int
column_values = data[['ST_COLCODE']].values.ravel()
unique_values =  pd.unique(column_values)
unique_values
data['ST_COLCODE']=data['ST_COLCODE'].astype(int)

column_values = data[['ST_COLCODE']].values.ravel()
unique_values =  pd.unique(column_values)
unique_values.sort()
unique_values
### But some are invalid values which are not in the metadata 

greaterthan_84_index=data[data['ST_COLCODE']>84].index
data.drop(greaterthan_84_index,inplace=True)
invalid_st_colcode_index=data[data['ST_COLCODE']==6].index
data.drop(invalid_st_colcode_index,inplace=True)
data.shape
invalid_st_colcode_index=data[data['ST_COLCODE']==7].index
data.drop(invalid_st_colcode_index,inplace=True)
data.shape
invalid_st_colcode_index=data[data['ST_COLCODE']==8].index
data.drop(invalid_st_colcode_index,inplace=True)
data.shape
invalid_st_colcode_index=data[data['ST_COLCODE']==9].index
data.drop(invalid_st_colcode_index,inplace=True)
data.shape
invalid_st_colcode_index=data[data['ST_COLCODE']==17].index
data.drop(invalid_st_colcode_index,inplace=True)
data.shape
invalid_st_colcode_index=data[data['ST_COLCODE']==18].index
data.drop(invalid_st_colcode_index,inplace=True)
data.shape
invalid_st_colcode_index=data[data['ST_COLCODE']==33].index
data.drop(invalid_st_colcode_index,inplace=True)
data.shape
#33-39 is not present in dataset
invalid_st_colcode_index=data[data['ST_COLCODE']==58].index
data.drop(invalid_st_colcode_index,inplace=True)
data.shape
#58,59 also not present
#68,69,70 also no present
#75-80 also not present

### UNDER_INFL has both N,Y and 0,1 
data['UNDERINFL'].value_counts()
data
data=data.replace(to_replace=['Y','N'],value=[1,0])
data=data.replace(to_replace=['1','0'],value=[1,0])
data['UNDERINFL'].value_counts()<jupyter_output><empty_output><jupyter_text>WE NEED TO CHANGE VALUE 1 TO 0 && 2to 1 in SEVERITYCODE for preparing for logistic regression !!!1 --> INJURY collision0 --> Property damage only collsion  WARNING : EXECUTE THE CODE BELOW EXACTLY ONCE ONLY !!!!<jupyter_code>data[['SEVERITYCODE']]=data[['SEVERITYCODE']].replace(1,0)
data[['SEVERITYCODE']]=data[['SEVERITYCODE']].replace(2,1)
data
# WE Transform like this because we need to measure % of chance of injury which is easier after we do such transformation
data.describe(include='all')<jupyter_output><empty_output><jupyter_text>We are moving on to deriving some useful insights by grouping data!<jupyter_code>data.groupby('ADDRTYPE').mean()<jupyter_output><empty_output><jupyter_text>we can understand that accidents at intersection are more inury prone & accidents at alley are mostly property damage only colllsion!  accidents in alley typically involved less persons lesser influence of alcohol in accidents at intersections  parked cars suffer much more damage in block than other 2 ollsions in alley are very rare compared to other 2 !less influence of alcohol in Intersection accidents Accidents involving parked cars far occur more in Block  More pedestrians are involoved in Intersection  accidents and very less in block !<jupyter_code>data.groupby('COLLISIONTYPE').mean()
according to data pedestrian and cycle are involved in more injury prone and less property only damages 
### Parked car and Sideswipe usualyy result in property only damages ! 
### MOst accidents typically involve 2-3 persons !
### NO head on collison involved pedestrians
### Less alcohol influence involved in Left and Right Turn accidents
data.groupby('PERSONCOUNT').mean()
NO influence of alcohol in accidents involving 13 or more people !
No parked cars where hit in accidents involving 10 or more people
NO pedestrian was involved in accidents involving 18 or more people (except 1 pedestrian in exactly 1 accident involving exactly 32 people on clear daytime in dry road !)
Only property damage was observed in accidents involving 57,53,47,44,43,41,36,35,31,26,21 people
No property and only injuries in accidents involving 81,54,48,39 people
data.groupby('PEDCOUNT').mean()
No property was damaged in accidents involving 4 or more pedestrians 
### Very high property damage in accidents not invloving pedestrians 
### When pedestrians are involved , parked cars are rarely hit 
data.groupby('VEHCOUNT').mean()
Injury only accidents involving 10 vehicles and Property only damage in accidents involving 12 vehicles
### Almost equal probability of Injury/Property Damage in accidents involving 5,7 or 11 vehicles
### NO pedestrians involved in accidents involving 8 or more vehicles 
### No influence of alcohol in accidnts involving 10 or 12 vehicles
### No parked Cars hit in accidents with more than 10 vehicles 
### Person count typically increases with vehicle count almost linearly 

data.groupby('UNDERINFL').mean()
UNDERINFL 

### ALmost similar probability of injury/property only damage whether under influence of alcohol or not , even though very less amount of cases under influence of alcohol(9039 vs 175439)
### Mostly involve 2or3 persons and mostly 2 vehicles whether under alcohol or not 
### Parked cars hit more often under influence of alcohol 

data.groupby('WEATHER').mean()

### Unknown cases mostly result in property damage 
### MOre injury prone accidents occur in Partly cloudy weather 
### MOstly 2or3 persons and 1or2 vehicles involved in accident whatever the weather 
### NO pedestrians involved in accidents in Partly cloudy or Severe crosswind weather 
### Most pedestrians involved in Hail/FReezing rain weather accidents 
### NO accidents under alcohol influence in partly cloudy weather and most in severe crosswind weather 
### very less no. of parked cars hit in rain !
data.groupby('ROADCOND').mean()
MOre property damage in UNknown road conditions and comparitively more injuries in Oil road condition
### 2-3 persons and 1-2 vehicles are typically involved here as well! 
### No pedestrain accident in oil road condition and more number of pedestrian accident in Dry roads
### No alcohol influenced accidents in Oil road and most in Sand/Dirt/MUd roads 
### No parked cars were hit in oil and standing water road condition while most in unknown/Snowy conditions

data.groupby('LIGHTCOND').mean()
More property damage in unknown and Drk-NOstreet lights condition
### NOalcohol influenced accidents in DARK-UNknown LIghting AND most in Dark-STreet lights on condition
### More parked cars hit in Dark-Unknown lighting

data.groupby('ST_COLCODE').mean()
Unicycle,tricycle and domestic animal(horse,cow,sheep) not involved in any accident
Vehicle Struck By State Road or ConstructionMachinery,62 Vehicle Hits County Road or ConstructionMachinery,63 Vehicle Struck By County Road or ConstructionMachinery not involved in any accidnt!
No property damage in accidnts 0f 49 st_colcode Non Domestic Animal (deer, bear, elk, etc)
No pedestrian involved in categories 73-84,57-67 , 40,41,29,24,25
No alcohol influence in Vehicles hitting train accidents
no parked cars hit in valid categories 60 to 84

data.groupby('HITPARKEDCAR').mean()
More property damage in accidents involving parked cars and less not involving parked cars
### pedestrains are very less involved in parked car accidents
### Alcohol influence in hitting parked car is slightly higher than accidents not involving parked car !

data.head()
# WE MAKE DICTIONARIES FOR STRING THE ENCODING OF DIFFERENT CATEGORICAL VALUES !
addrtype_dict={'Block':1,'Intersection':2,'Alley':3}
collisiontype_dict={'Parked Car':1,'Angles':2,'Rear Ended':3,'Other':4,'Sideswipe':5,'Left Turn':6,'Pedestrian':7,'Cycles':8,'Right Turn':9,'Head On':10}
weather_dict={'Clear':1,'Raining':2,'Overcast':3,'Unknown':4,'Snowing':5,'Other':6,'Fog/Smog/Smoke':7,'Sleet/Hail/Freezing Rain':8,'Blowing Sand/Dirt':9,'Severe Crosswind':10,'Partly Cloudy':11}
roadcond_dict={'Dry':1,'Wet':2,'Unknown':3,'Ice':4,'Snow/Slush':5,'Other':6,'Standing Water':7,'Sand/Mud/Dirt':8,'Oil':9}
lightcond_dict={'Daylight':1,'Dark - Street Lights On':2,'Unknown':3,'Dusk':4,'Dawn':5,'Dark - No Street Lights':6,'Dark - Street Lights Off':7,'Other':8,'Dark - Unknown Lighting':9}
clean_up_categorical={'ADDRTYPE':addrtype_dict ,'COLLISIONTYPE':collisiontype_dict ,'WEATHER':weather_dict ,'ROADCOND':roadcond_dict ,'LIGHTCOND':lightcond_dict}<jupyter_output><empty_output><jupyter_text>LABEL ENCODING<jupyter_code>data1=data.copy()

obj_data1 = data1.select_dtypes(include=['object']).copy()
obj_data1.head()
#obj_data1[obj_data1.isnull().any(axis=1)]
obj_data1.replace(clean_up_categorical, inplace=True)
#obj_data1.head(50)
obj_data1.rename(columns={'ADDRTYPE':'ADDRTYPENUM','COLLISIONTYPE':'COLLISIONTYPENUM','WEATHER':'WEATHERNUM','ROADCOND':'ROADCONDNUM','LIGHTCOND':'LIGHTCONDNUM' },inplace=True)
obj_data1.head(50)
result=pd.concat([data1,obj_data1],axis=1,sort=False)
result<jupyter_output><empty_output><jupyter_text>Here we created additional 5 columns which correspond to the numerical values of the Categorical variables in the 5 columns specified in the dictionary<jupyter_code>#we can now easily build the model with selected variables which are numbers only !

data=result<jupyter_output><empty_output><jupyter_text>DATA VISUALIZATION !<jupyter_code>sns.countplot(x='SEVERITYCODE',data=data)<jupyter_output><empty_output><jupyter_text>More accidents involve property damage than collisions !!<jupyter_code>sns.heatmap(data.isnull())<jupyter_output><empty_output><jupyter_text>NO null values are present after data cleaning !<jupyter_code>data.shape
data.shape
data.head(3)<jupyter_output><empty_output><jupyter_text>Model Building !<jupyter_code>X=np.asarray(data[['PERSONCOUNT','PEDCOUNT','VEHCOUNT','UNDERINFL','ST_COLCODE','HITPARKEDCAR','ADDRTYPENUM','COLLISIONTYPENUM','WEATHERNUM','ROADCONDNUM','LIGHTCONDNUM']])
X[0:5]
y = np.asarray(data['SEVERITYCODE'])
y[0:5]
from sklearn import preprocessing
X = preprocessing.StandardScaler().fit(X).transform(X)
X[0:5]<jupyter_output>/opt/conda/envs/Python36/lib/python3.6/site-packages/sklearn/utils/validation.py:595: DataConversionWarning: Data with input dtype int64 was converted to float64 by StandardScaler.
  warnings.warn(msg, DataConversionWarning)
/opt/conda/envs/Python36/lib/python3.6/site-packages/sklearn/utils/validation.py:595: DataConversionWarning: Data with input dtype int64 was converted to float64 by StandardScaler.
  warnings.warn(msg, DataConversionWarning)<jupyter_text>NOW split the dataset into train and test<jupyter_code>from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=4)
print ('Train set:', X_train.shape,  y_train.shape)
print ('Test set:', X_test.shape,  y_test.shape)<jupyter_output>Train set: (147332, 11) (147332,)
Test set: (36834, 11) (36834,)<jupyter_text>Modelling using Logistic Regression Lets build our model using LogisticRegression from Scikit-learn package. This function implements logistic regression and can use different numerical optimizers to find parameters, including ‘newton-cg’, ‘lbfgs’, ‘liblinear’, ‘sag’, ‘saga’ solvers. You can find extensive information about the pros and cons of these optimizers if you search it in internet.The version of Logistic Regression in Scikit-learn, support regularization. Regularization is a technique used to solve the overfitting problem in machine learning models. C parameter indicates inverse of regularization strength which must be a positive float. Smaller values specify stronger regularization. Now lets fit our model with train set:<jupyter_code>from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
LR = LogisticRegression(C=0.01, solver='liblinear').fit(X_train,y_train)
LR<jupyter_output><empty_output><jupyter_text>The class predictions are<jupyter_code>yhat = LR.predict(X_test)
yhat<jupyter_output><empty_output><jupyter_text>But we need %of chance of injury !! Instead , we use this<jupyter_code>yhat_prob = LR.predict_proba(X_test)
yhat_prob
print("PROBABILITY ARRAY = %.2f ",yhat_prob*100)<jupyter_output>PROBABILITY ARRAY = %.2f  [[86.61087514 13.38912486]
 [57.88473905 42.11526095]
 [69.90064155 30.09935845]
 ...
 [90.07845218  9.92154782]
 [53.92592449 46.07407551]
 [79.22570861 20.77429139]]<jupyter_text>Evaluation  jaccard indexLets try jaccard index for accuracy evaluation. we can define jaccard as the size of the intersection divided by the size of the union of two label sets. If the entire set of predicted labels for a sample strictly match with the true set of labels, then the subset accuracy is 1.0; otherwise it is 0.0.<jupyter_code>from sklearn.metrics import jaccard_similarity_score
jaccard_similarity_score(y_test, yhat)<jupyter_output><empty_output><jupyter_text>confusion matrixAnother way of looking at accuracy of classifier is to look at confusion matrix.<jupyter_code>from sklearn.metrics import classification_report, confusion_matrix
import itertools
def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion matrix, without normalization')

    print(cm)

    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    fmt = '.2f' if normalize else 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], fmt),
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
print(confusion_matrix(y_test, yhat, labels=[1,0]))
# Compute confusion matrix
cnf_matrix = confusion_matrix(y_test, yhat, labels=[1,0])
np.set_printoptions(precision=2)


# Plot non-normalized confusion matrix
plt.figure()
plot_confusion_matrix(cnf_matrix, classes=['churn=1','churn=0'],normalize= False,  title='Confusion matrix')
print (classification_report(y_test, yhat))<jupyter_output>precision    recall  f1-score   support

           0       0.73      0.96      0.83     25583
           1       0.67      0.20      0.31     11251

   micro avg       0.73      0.73      0.73     36834
   macro avg       0.70      0.58      0.57     36834
weighted avg       0.71      0.73      0.67     36834<jupyter_text>Based on the count of each section, we can calculate precision and recall of each label:Precision is a measure of the accuracy provided that a class label has been predicted. It is defined by: precision = TP / (TP + FP)Recall is true positive rate. It is defined as: Recall =  TP / (TP + FN)So, we can calculate precision and recall of each class.F1 score: Now we are in the position to calculate the F1 scores for each label based on the precision and recall of that label.The F1 score is the harmonic average of the precision and recall, where an F1 score reaches its best value at 1 (perfect precision and recall) and worst at 0. It is a good way to show that a classifer has a good value for both recall and precision.And finally, we can tell the average accuracy for this classifier is the average of the F1-score for both labels, which is 0.67 in our case.  LogLoss<jupyter_code>from sklearn.metrics import log_loss
log_loss(y_test, yhat_prob)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09306179568387715, 'fraction_numerical': 0.03270236429255358, 'mean_word_length': 5.497966020579086, 'pattern_counts': {'":': 0, '<': 104, '<?xml version=': 0, '>': 112, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '519064', 'license': 'MIT', 'n_tokens_mistral': 8861, 'n_tokens_neox': 8165, 'n_words': 2727, 'path': 'IBMcapstoneproject.ipynb', 'repo_name': 'sayanth98/Coursera_Capstone', 'size': '27158'}