<jupyter_start><jupyter_text>Performing Data Cleaning and AnalysisData Dictionary: -->**Variable        Description**1. Survived	- Survived (1) or died (0)2. Pclass -	Passenger’s class (1 = 1st, 2 = 2nd, 3 = 3rd)3. Name	- Passenger’s name4. Sex -	Passenger’s sex5. Age	- Passenger’s age6. SibSp -	Number of siblings/spouses aboard7. Parch -	Number of parents/children aboard (Some children travelled only with a nanny, therefore parch=0 for them.)8. Ticket -	Ticket number9. Fare -	Fare10. Cabin -	Cabin11. Embarked -	Port of embarkation (C = Cherbourg, Q = Queenstown, S = Southampton)<jupyter_code>"""
Deleting the columns not required for determining the survival of a person
"""

del train['PassengerId']
del train['Ticket']
del train['Fare']
del train['Cabin']
del train['Name']

train.head()
train.shape
train.describe()
train.isnull().sum()
"""Since there are ONLY 2 rows whose Embarked data is not known, 
therefore we can neglect those 2 rows as they will not make much of a difference"""


"""
Similarly, deleting the columns not required for determining the survival of a person for the data stored in test.csv
"""

del test['Ticket']
del test['Fare']
del test['Cabin']
del test['Name']

test.head()
test.describe()
test.isnull().sum()
def getNum(str):
    if str=='male':
        return '1'
    if str=='female':
        return '2'
train["Gender"]=train["Sex"].apply(getNum)
#We have created a new column called "Gender" and 
#filling it with values 1 ,2 based on the values of sex column
train.head()

test["Gender"]=test["Sex"].apply(getNum)
#We have created a new column called "Gender" and 
#filling it with values 1 ,2 based on the values of sex column
test.head()
del train['Sex']
del test['Sex']
train.head()
test.head()
train[['Pclass','Survived']].groupby(['Pclass']).mean().sort_values(by='Survived',ascending=True)
train[['Gender','Survived']].groupby(['Gender']).mean().sort_values(by='Survived',ascending=True)
train[['SibSp','Survived']].groupby(['SibSp']).mean().sort_values(by='Survived', ascending=True)
train[['Parch','Survived']].groupby(['Parch']).mean().sort_values(by='Survived', ascending=True)<jupyter_output><empty_output><jupyter_text>Analyzing Data by VisualizationInorder to understand who would have had a better probability of survival, we should visualize the patients who survived based on age, passenger class and etc.<jupyter_code>sns.countplot(train['Survived'])

age_hist = sns.FacetGrid(train, col='Survived')
age_hist.map(plt.hist, 'Age')
age_hist.set_ylabels('Number')<jupyter_output><empty_output><jupyter_text>Among those who didn't survive, most people were of the age 20-30 years whereas among those who survived, the number of infants are much higher than teenagers.  Number of people who survived over age and passenger class<jupyter_code>pclass_age_grid = sns.FacetGrid(train, col='Survived', row='Pclass', height=2.0, aspect=1.6)
pclass_age_grid.map(plt.hist, 'Age', alpha=0.5, bins=20)
pclass_age_grid.add_legend()
pclass_age_grid.set_ylabels('Number')<jupyter_output><empty_output><jupyter_text>Handling the NULL values in the column Age We have NULL value inplace of Age for some of the people in both training and testing data. So, one way is to fill them with the mean values i.e. fill the ones who have survived with the mean age of the survived people and similarly fill those who haven't survived with the mean age of all non-survived people. But, this will only solve the problem for the training data and not for the testing data as we have to predict their survival status. So, we can tackle the issue by creating an array that contains random numbers, which are computed based on the mean age value in regards to the standard deviation and is_null.<jupyter_code>#finding mean survived age
mean_sur= train[train.Survived==1]['Age'].mean()
mean_sur
# Finding the mean age of "Not Survived" people
mean_nsur=train[train['Survived']==0]['Age'].mean()
mean_nsur
data = [train, test]

for dataset in data:
    mean = train["Age"].mean()
    std = test["Age"].std()
    is_null = dataset["Age"].isnull().sum()
    
    # compute random numbers between the mean, std and is_null
    rand_age = np.random.randint(mean - std, mean + std, size = is_null)
    
    # fill NaN values in Age column with random values generated
    age_slice = dataset["Age"].copy()
    age_slice[np.isnan(age_slice)] = rand_age
    dataset["Age"] = age_slice
    dataset["Age"] = train["Age"].astype(int)
    
    
train["Age"].isnull().sum()

#Renaming "gender" column
train.rename(columns={'Gender':'Sex'}, inplace=True)

test.rename(columns={'Gender':'Sex'}, inplace=True)

#Removing the 2 rows having null value for Embarked column
train.dropna(inplace=True)

train.head()
train.info()
#Combining 'Parch' & 'SibSp' as 'Realtives'
train['Family_Members']=train['Parch']+train['SibSp'] + 1
test['Family_Members']=test['Parch']+test['SibSp'] + 1

del train['SibSp']
del train['Parch']

del test['SibSp']
del test['Parch']
train.head()
test.head()<jupyter_output><empty_output><jupyter_text>Grouping the Age data The age groups need to be converted into different sub-groups so that better prediction model can be formed<jupyter_code>data=[train,test]

for dataset in data:
    dataset.loc[ dataset['Age'] <= 10, 'Age'] = 0
    dataset.loc[(dataset['Age'] > 10) & (dataset['Age'] <= 20), 'Age'] = 1
    dataset.loc[(dataset['Age'] > 20) & (dataset['Age'] <= 25), 'Age'] = 2
    dataset.loc[(dataset['Age'] > 25) & (dataset['Age'] <= 30), 'Age'] = 3
    dataset.loc[(dataset['Age'] > 30) & (dataset['Age'] <= 37), 'Age'] = 4
    dataset.loc[(dataset['Age'] > 37) & (dataset['Age'] <= 45), 'Age'] = 5
    dataset.loc[ dataset['Age'] > 45 , 'Age'] = 6

train['Age'].value_counts()
train.head()
test.head()
data=[train,test]

for dataset in data:
    dataset['Embarked']=dataset['Embarked'].map({'C': 0, 'S': 1, 'Q': 2}).astype(int)
    
train.head()
train[['Age','Survived']].groupby(['Age']).mean().sort_values(by='Survived', ascending=True)
train[['Embarked','Survived']].groupby(['Embarked']).mean().sort_values(by='Survived', ascending=True)
train[['Pclass','Survived']].groupby(['Pclass']).mean().sort_values(by='Survived', ascending=True)
sns.heatmap(train.corr(), annot=True)
sns.heatmap(test.corr(), annot=True)
train

test<jupyter_output><empty_output><jupyter_text>Building Models for Prediction Now we will train multiple Machine Learning algorithms over the training data to predict the survival on our testing data and analyze the results thus obtained. We might also use use cross-validation in the end.  We know that since the survival is represented as either 0 or 1, therefore it is a Classification problem. The algorithms used for the same are: Logistic Regression Support Vector Machines KNN or K-Nearest Neighbors Decision Trees Random Forest Stochastic Gradient descent (SGD) Gaussian Naive Bayes<jupyter_code>X_train= train.drop(['Survived'], axis =1)
y_train= train['Survived']

X_test=test.drop('PassengerId', axis=1).copy()
X_test.shape<jupyter_output><empty_output><jupyter_text>Logistic Regression<jupyter_code>log_reg = LogisticRegression()
log_reg.fit(X_train, y_train)

y_pred = log_reg.predict(X_test)
print(y_pred)

round(np.mean(y_pred), 2)
# Checking accuracy of the linear regressor model

log_reg_acc = round(log_reg.score(X_train,y_train)*100, 2)
print(log_reg_acc,'%')<jupyter_output>79.64 %<jupyter_text>We can find the correlation between the different features within the data inorder to analyze the survival of the people onboard.<jupyter_code>df_coeff= pd.DataFrame(train.columns.delete(0))
df_coeff.columns = ['Feature']
df_coeff['Correlation'] = pd.Series(log_reg.coef_[0])

df_coeff.sort_values(by='Correlation', ascending=False)<jupyter_output><empty_output><jupyter_text>From this we can conclude that Females (Sex=2) have a higher probability of surviving.  Cross-Validation Cross-Validation protects against overfitting. It is a resampling method which tells us how well our model would generalize to unseen data. This is achieved by fixing a number of partitions of the dataset called folds, predicting each fold separately, and averaging the predictions in the end.<jupyter_code># Set our robust cross-validation scheme!
kf = KFold(n_splits = 10, random_state = 2)

# Print CV accuracy estimate:
#print(cross_val_score(logisticRegression, X_test, y_test, cv = kf).mean())
scores= cross_val_score(log_reg, X_train, y_train, cv = kf, scoring='accuracy')

mean_acc_log = scores.mean()*100

print('Scores: ', scores*100, '%')
print('Mean: ', mean_acc_log, '%')
print('Standard Deviation: ', scores.std()*100, '%\n')

pred= cross_val_predict(log_reg, X_train, y_train, cv=kf)
print('Confusion Matrix: \n' ,confusion_matrix(y_train, pred),'\n')

print("Precision: ", round(precision_score(y_train, pred)*100, 2),'%')
print("Recall: ", round(recall_score(y_train, pred)*100, 2), '%')
print('F1 Score: ', round(f1_score(y_train, pred)*100, 2), '%')<jupyter_output>Scores:  [74.15730337 80.8988764  75.28089888 78.65168539 77.52808989 76.40449438
 77.52808989 75.28089888 85.39325843 81.81818182] %
Mean:  78.29417773237999 %
Standard Deviation:  3.3127923441226788 %

Confusion Matrix: 
 [[468  81]
 [112 228]] 

Precision:  73.79 %
Recall:  67.06 %
F1 Score:  70.26 %<jupyter_text>The first row is about the not-survived-predictions: 470 passengers were correctly classified as not survived (called true negatives) and 79 where wrongly classified as not survived (false positives). The second row is about the survived-predictions: 110 passengers where wrongly classified as survived (false negatives) and 230 where correctly classified as survived (true positives).  Support Vector Machine (SVM)<jupyter_code>svc = SVC()
svc.fit(X_train, y_train)

y_pred = svc.predict(X_test)
print(y_pred)
y_pred.shape
round(np.mean(y_pred), 2)
# Checking accuracy of the support vector model

svc_acc = round(svc.score(X_train,y_train)*100, 2)
print(svc_acc,'%')
# Print CV accuracy estimate:
#print(cross_val_score(SVC(), X_test, y_test, cv = kf).mean())

scores= cross_val_score(SVC(), X_train, y_train, cv = kf, scoring='accuracy')

mean_acc_svc = scores.mean()*100

print('Scores: ', scores*100, '%')
print('Mean: ', mean_acc_svc, '%')
print('Standard Deviation: ', scores.std()*100, '%\n')

pred= cross_val_predict(svc, X_train, y_train, cv=kf)
print('Confusion Matrix: \n' ,confusion_matrix(y_train, pred),'\n')

print("Precision: ", round(precision_score(y_train, pred)*100, 2),'%')
print("Recall: ", round(recall_score(y_train, pred)*100, 2), '%')
print('F1 Score: ', round(f1_score(y_train, pred)*100, 2), '%')<jupyter_output>Scores:  [79.7752809  86.51685393 78.65168539 84.26966292 82.02247191 82.02247191
 79.7752809  77.52808989 88.76404494 85.22727273] %
Mean:  82.45531154239019 %
Standard Deviation:  3.4750078757718716 %

Confusion Matrix: 
 [[491  58]
 [ 98 242]] 

Precision:  80.67 %
Recall:  71.18 %
F1 Score:  75.62 %<jupyter_text>The first row is about the not-survived-predictions: 493 passengers were correctly classified as not survived (called true negatives) and 56 where wrongly classified as not survived (false positives). The second row is about the survived-predictions: 100 passengers where wrongly classified as survived (false negatives) and 240 where correctly classified as survived (true positives).  K-Nearest Neighbor (KNN)<jupyter_code>knn = KNeighborsClassifier()
knn.fit(X_train, y_train)

y_pred= knn.predict(X_test)
print(y_pred)
round(np.mean(y_pred), 2)
# Checking accuracy of the knn model

knn_acc = round(knn.score(X_train,y_train)*100, 2)
print(knn_acc,'%')
# Print CV accuracy estimate:
#print(cross_val_score(KNeighborsClassifier(), X_test, y_test, cv = kf).mean())

scores= cross_val_score(KNeighborsClassifier(), X_train, y_train, cv = kf, scoring='accuracy')

mean_acc_knn = scores.mean()*100

print('Scores: ', scores*100, '%')
print('Mean: ', mean_acc_knn, '%')
print('Standard Deviation: ', scores.std()*100, '%\n')

pred= cross_val_predict(knn, X_train, y_train, cv=kf)
print('Confusion Matrix: \n' ,confusion_matrix(y_train, pred), '\n')

print("Precision: ", round(precision_score(y_train, pred)*100, 2),'%')
print("Recall: ", round(recall_score(y_train, pred)*100, 2), '%')
print('F1 Score: ', round(f1_score(y_train, pred)*100, 2), '%')<jupyter_output>Scores:  [75.28089888 79.7752809  78.65168539 82.02247191 86.51685393 75.28089888
 79.7752809  76.40449438 83.14606742 78.40909091] %
Mean:  79.52630234933606 %
Standard Deviation:  3.4078007011732017 %

Confusion Matrix: 
 [[481  68]
 [114 226]] 

Precision:  76.87 %
Recall:  66.47 %
F1 Score:  71.29 %<jupyter_text>The first row is about the not-survived-predictions: 484 passengers were correctly classified as not survived (called true negatives) and 65 where wrongly classified as not survived (false positives). The second row is about the survived-predictions: 107 passengers where wrongly classified as survived (false negatives) and 233 where correctly classified as survived (true positives).  Decision Trees<jupyter_code>tree = DecisionTreeClassifier()
tree.fit(X_train, y_train)

y_pred = tree.predict(X_test)
print(y_pred)
round(np.mean(y_pred), 2)
#Checking accuracy of Decision Tree model

tree_acc = round(tree.score(X_train, y_train)*100, 3)
print(tree_acc,'%')
# Print CV accuracy estimate:
#print(cross_val_score(DecisionTreeClassifier(), X_test, y_test, cv = kf).mean())

scores= cross_val_score(DecisionTreeClassifier(), X_train, y_train, cv = kf, scoring='accuracy')

mean_acc_tree = scores.mean()*100

print('Scores: ', scores*100, '%')
print('Mean: ', mean_acc_tree, '%')
print('Standard Deviation: ', scores.std()*100, '%\n')<jupyter_output>Scores:  [73.03370787 82.02247191 79.7752809  76.40449438 80.8988764  76.40449438
 76.40449438 74.15730337 86.51685393 76.13636364] %
Mean:  78.17543411644536 %
Standard Deviation:  3.8794954123243657 %<jupyter_text>Random Forest<jupyter_code>random_forest = RandomForestClassifier(n_estimators=100)
random_forest.fit(X_train, y_train)

y_pred = random_forest.predict(X_test)
print(y_pred)

round(np.mean(y_pred),2 )
# Checking accuracy of the Random Forest Classifier

random_forest.score(X_train, y_train)
acc_random_forest = round(random_forest.score(X_train, y_train) * 100, 3)
print(acc_random_forest,'%')
# Print CV accuracy estimate:
#print(cross_val_score(RandomForestClassifier(), X_test, y_test, cv = kf).mean())

scores= cross_val_score(RandomForestClassifier(), X_train, y_train, cv = kf, scoring='accuracy')

mean_acc_ranfor = scores.mean()*100

print('Scores: ', scores*100, '%')
print('Mean: ', mean_acc_ranfor, '%')
print('Standard Deviation: ', scores.std()*100, '%\n')

pred= cross_val_predict(random_forest, X_train, y_train, cv=kf)
print('Confusion Matrix: \n' ,confusion_matrix(y_train, pred), '\n')

print("Precision: ", round(precision_score(y_train, pred)*100, 2),'%')
print("Recall: ", round(recall_score(y_train, pred)*100, 2), '%')
print('F1 Score: ', round(f1_score(y_train, pred)*100, 2), '%')<jupyter_output>Scores:  [74.15730337 77.52808989 82.02247191 75.28089888 82.02247191 78.65168539
 78.65168539 74.15730337 87.64044944 80.68181818] %
Mean:  79.07941777323799 %
Standard Deviation:  3.985445770810656 %

Confusion Matrix: 
 [[477  72]
 [115 225]] 

Precision:  75.76 %
Recall:  66.18 %
F1 Score:  70.64 %<jupyter_text>The first row is about the not-survived-predictions: 479 passengers were correctly classified as not survived (called true negatives) and 70 where wrongly classified as not survived (false positives). The second row is about the survived-predictions: 115 passengers where wrongly classified as survived (false negatives) and 225 where correctly classified as survived (true positives).  Stochastic Gradient Descent (SGD)<jupyter_code>sgd = SGDClassifier(max_iter=5, tol=None)
sgd.fit(X_train, y_train)

y_pred = sgd.predict(X_test)
print(y_pred)
round(np.mean(y_pred), 2)
#Checking accuracy of the SGD model

sgd.score(X_train, y_train)
acc_sgd = round(sgd.score(X_train, y_train) * 100, 2)
print(acc_sgd,'%')
# Print CV accuracy estimate:
#print(cross_val_score(SGDClassifier(max_iter=5, tol=None), X_test, y_test, cv = kf).mean())

scores= cross_val_score(SGDClassifier(max_iter=5, tol=None), X_train, y_train, cv = kf, scoring='accuracy')

mean_acc_sgd = scores.mean()*100

print('Scores: ', scores*100, '%')
print('Mean: ', mean_acc_sgd, '%')
print('Standard Deviation: ', scores.std()*100, '%\n')

pred= cross_val_predict(sgd, X_train, y_train, cv=kf)
print('Confusion Matrix: \n' ,confusion_matrix(y_train, pred), '\n')

print("Precision: ", round(precision_score(y_train, pred)*100, 2),'%')
print("Recall: ", round(recall_score(y_train, pred)*100, 2), '%')
print('F1 Score: ', round(f1_score(y_train, pred)*100, 2), '%')<jupyter_output>Scores:  [67.41573034 77.52808989 66.29213483 74.15730337 74.15730337 69.66292135
 78.65168539 71.91011236 78.65168539 80.68181818] %
Mean:  73.9108784473953 %
Standard Deviation:  4.756912892690448 %

Confusion Matrix: 
 [[458  91]
 [167 173]] 

Precision:  65.53 %
Recall:  50.88 %
F1 Score:  57.28 %<jupyter_text>The first row is about the not-survived-predictions: 458 passengers were correctly classified as not survived (called true negatives) and 91 where wrongly classified as not survived (false positives). The second row is about the survived-predictions: 16 passengers where wrongly classified as survived (false negatives) and 224 where correctly classified as survived (true positives).  Gaussian Naive Bayes<jupyter_code>gaussian = GaussianNB() 
gaussian.fit(X_train, y_train)  

y_pred = gaussian.predict(X_test)
print(y_pred)

round(np.mean(y_pred), 2)
#Checking accuracy for the Gaussian Naive Bayes model

acc_gaussian = round(gaussian.score(X_train, y_train) * 100, 2)
print(acc_gaussian,'%')
# Print CV accuracy estimate:
#print(cross_val_score(GaussianNB() , X_test, y_test, cv = kf).mean())

scores= cross_val_score(GaussianNB() , X_train, y_train, cv = kf, scoring='accuracy')

mean_acc_gau = scores.mean()*100

print('Scores: ', scores*100, '%')
print('Mean: ', mean_acc_gau, '%')
print('Standard Deviation: ', scores.std()*100, '%\n')

#Confusion Matrix
pred= cross_val_predict(gaussian, X_train, y_train, cv=kf)
print('Confusion Matrix: \n' ,confusion_matrix(y_train, pred), '\n')

print("Precision: ", round(precision_score(y_train, pred)*100, 2),'%')
print("Recall: ", round(recall_score(y_train, pred)*100, 2), '%')
print('F1 Score: ', round(f1_score(y_train, pred)*100, 2), '%')<jupyter_output>Scores:  [75.28089888 77.52808989 77.52808989 82.02247191 77.52808989 78.65168539
 79.7752809  80.8988764  83.14606742 81.81818182] %
Mean:  79.41777323799796 %
Standard Deviation:  2.3912633037949353 %

Confusion Matrix: 
 [[464  85]
 [ 98 242]] 

Precision:  74.01 %
Recall:  71.18 %
F1 Score:  72.56 %<jupyter_text>The first row is about the not-survived-predictions: 464 passengers were correctly classified as not survived (called true negatives) and 85 where wrongly classified as not survived (false positives). The second row is about the survived-predictions: 98 passengers where wrongly classified as survived (false negatives) and 242 where correctly classified as survived (true positives).  Finding the Best Model<jupyter_code>results = pd.DataFrame({
    'Model': ['Logistic Regression', 'Support Vector Machines', 'KNN',  
              'Decision Tree','Random Forest',
              'Stochastic Gradient Decent', 
              'Gaussian Naive Bayes'],
    'Score': [log_reg_acc, svc_acc, knn_acc,  
              tree_acc, acc_random_forest, acc_sgd,
              acc_gaussian],
    'Mean Score': [mean_acc_log, mean_acc_svc, mean_acc_knn,
                   mean_acc_tree, mean_acc_ranfor, 
                   mean_acc_sgd, mean_acc_gau]})
df_result = results.sort_values(by='Mean Score', ascending=False)
df_result = df_result.set_index('Model')
df_result<jupyter_output><empty_output><jupyter_text>As we can see, the Decision Tree Classifer & Random Forest Classifier coincidently outputs an equal amount of 86.73% accuracy on the test set and are both the most successful models for the set of features and the model that we have built. But, we also know that just calculating the accuracy without cross-validating could result in Overfitting that we can easily see in our models. So, after performing cross-validation, we find out the mean accuracy of each of the model which correctly identifies Support Vector Machines (SVC) as the best model with an accuracy of 82.45% for the given dataset.<jupyter_code>importances=pd.DataFrame({'Features': X_train.columns, 'Importance':np.round(random_forest.feature_importances_, 3)})
importances = importances.sort_values('Importance',ascending=False).set_index('Features')
print(importances.head())
importances.plot.bar()
importances=pd.DataFrame({'Features': X_train.columns, 'Importance':np.round(tree.feature_importances_, 3)})
importances = importances.sort_values('Importance',ascending=False).set_index('Features')
print(importances.head())
importances.plot.bar()
test<jupyter_output><empty_output><jupyter_text>SUBMISSION<jupyter_code>submission = pd.DataFrame({
        'PassengerId': test['PassengerId'],
        'Survived': y_pred
    })
submission.to_csv('submission.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13385049365303245, 'fraction_numerical': 0.06666666666666667, 'mean_word_length': 5.976385700229583, 'pattern_counts': {'":': 0, '<': 71, '<?xml version=': 0, '>': 72, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '195132', 'license': 'MIT', 'n_tokens_mistral': 8391, 'n_tokens_neox': 7233, 'n_words': 1976, 'path': 'Machine Learning and Data Science/Titanic_Survival_Prediction/Titanic Survival Analysis.ipynb', 'repo_name': 'Brabec/Hacktoberfest2020', 'size': '21270'}