<jupyter_start><jupyter_text>Name: <NAME> Course: DSCI_633 Assignment 03  STEP 1: Set Up<jupyter_code>import sys
import os
import pandas as pd
import matplotlib.pyplot as plt
import sklearn
import seaborn as sns
import warnings
warnings.filterwarnings("ignore")
PROJECT_ROOT_DIR = "."
ASSIGNMENT_ID = "Assignment_03"
IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, ASSIGNMENT_ID, "images")

if not os.path.isdir(IMAGES_PATH):
    os.makedirs(IMAGES_PATH)
    
def save_fig(fig_id, tight_layout=True, fig_extension="png", resolution=300):
    path = os.path.join(IMAGES_PATH, fig_id + "." + fig_extension)
    print("Saving figure", fig_id)
    if tight_layout:
        plt.tight_layout()
    plt.savefig(path, format=fig_extension, dpi=resolution)<jupyter_output><empty_output><jupyter_text>STEP 2: Load the data<jupyter_code>def load_data(dataset_path):
    return pd.read_csv(dataset_path)

train_dataset_path = os.path.join(PROJECT_ROOT_DIR, ASSIGNMENT_ID, "titanic", "train.csv")
test_dataset_path = os.path.join(PROJECT_ROOT_DIR, ASSIGNMENT_ID, "titanic", "test.csv")
df = load_data(train_dataset_path)
df.head()<jupyter_output><empty_output><jupyter_text>STEP 3: Data Manipulation and Analysis Using Pandas Here the column `Survived` is the target and the remaining columns are the features. Let us make the target as the final column<jupyter_code>def change_column_list(dataframe, column_name, new_index):
    column_names = list(dataframe.columns)
    old_index = column_names.index(column_name)
    column_names.pop(old_index)
    column_names.insert(new_index, column_name)
    
#     column_names[new_index] = column_name
    return dataframe[column_names]

df = change_column_list(df, "Survived", len(df.columns))
df.head()
df.shape<jupyter_output><empty_output><jupyter_text>There are 891 rows and 12 columns in the data<jupyter_code>#Let's check for null values
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 12 columns):
PassengerId    891 non-null int64
Pclass         891 non-null int64
Name           891 non-null object
Sex            891 non-null object
Age            714 non-null float64
SibSp          891 non-null int64
Parch          891 non-null int64
Ticket         891 non-null object
Fare           891 non-null float64
Cabin          204 non-null object
Embarked       889 non-null object
Survived       891 non-null int64
dtypes: float64(2), int64(5), object(5)
memory usage: 83.7+ KB<jupyter_text>The Age and Cabin columns have majority of the null values<jupyter_code># Checking the statistics of the data
df.describe()<jupyter_output><empty_output><jupyter_text>Step 4: Discover and visualize the data to gain insights<jupyter_code>#Let's check the number of Passenger classes
df['Pclass'].unique()<jupyter_output><empty_output><jupyter_text>So there were 3 classes - First Class, Second class and third class<jupyter_code># Let's check the number of passengers in each class
pclass_Bar_plot = df["Pclass"].value_counts().plot.bar();
pclass_Bar_plot.set_ylabel("No of Passengers")
pclass_Bar_plot.set_xlabel("Passenger class")
pclass_Bar_plot.set_xticklabels(("Third Class", "First Class", "Second Class"))
pclass_Bar_plot.set_title("Count of Passengers in each class");<jupyter_output><empty_output><jupyter_text>It can be seen that there were close to 500 passengers in Third Class and a little over and below 200 in First and Second class respectively.<jupyter_code>#Let's check, how many of the passengers from each of these three classes survived 
c = pd.DataFrame(df.groupby(["Pclass", "Survived"])["Survived"].count().unstack().unstack().reset_index())
c.rename(columns={0:"count of dead and survived"}, inplace=True)
sns.factorplot("Pclass", "count of dead and survived", hue="Survived", data=c, kind="bar");<jupyter_output><empty_output><jupyter_text>- We can see that out of the passengers who survived, around 140 were from First Class, around 120 were from Third Class and around 90 were from the Third Class. - It can be seen that maximum passengers who did not survive were from the Third Class. The number count goes over 350- So if a passenger was from the Third Class, chances are very high that he did not survive.- For a passenger from the Second Class, the odds of survival/death rate are almost 50% since the count for passengers in Second Class who survived and did not survive is almost the same.- Also, we can say that for First Class, the survival rate is around 60% and the non-survival rate is around 40%. So, chances of survival for a First class passenger are higher.<jupyter_code># now let's check the nmber of males and females from those who survived.
maf = pd.DataFrame(df.groupby(["Sex", "Survived"])["Survived"].count().unstack().unstack().reset_index())
maf.rename(columns={0:"Survival/Non-survival count"}, inplace=True)
sns.factorplot("Sex", "Survival/Non-survival count", hue="Survived", data=maf, kind="bar");<jupyter_output><empty_output><jupyter_text>- From the above graph, we can say that the count of the non-survived males is ~480 and for males who survived is ~100. So thesurvival and non-survival rate is around 20% and 80% respectively. That means, males have a high probability of non-survival.- But for females, the survival rate seems quite high i.e. to be around 75% and the non-survival rate seems to be around 25%.<jupyter_code># Let's see from which class were the males and females.
mafc = pd.DataFrame(df.groupby(["Pclass", "Sex", "Survived"])["Survived"].agg("count"))
mafc.rename(columns={"Survived":"Survival/Non-Survival Count"}, inplace=True)
mafc<jupyter_output><empty_output><jupyter_text>- In the above table:    - 1, 2 and 3 in Pclass represent First Class, Second Class and Third Class respectively.     - 0, 1 in Survived represent Dead and Survived respectively. Females        - It can be seen that there were 94 females in total in 1st Class and from these 91 survived.     - There were 76 females in 2nd class and from these 70 survived.    - There were 144 females in 3rd class and out of them exactly 50% ie 7 survived.    - So if a person was a female in 1st class or 2nd class, chances of her survival would have been almost certain. Males    - The count of the no of males who did not survive increases non-linearly as we go from 1st class to 3rd class and the survival rate decreases.     - If the passenger is a male and he belongs to 2nd or 3rd class, his rate of survival seems very low.<jupyter_code># Let's inspect the Age feature now.
print("Max Age of Passengers: ",max(df["Age"]))
print("Min Age of Passengers: ",min(df["Age"]))
print("Mean Age of Passengers: ",round(df['Age'].mean(), 3))
sns.kdeplot(df['Age']);<jupyter_output>Max Age of Passengers:  80.0
Min Age of Passengers:  0.42
Mean Age of Passengers:  29.699<jupyter_text>- We can see that the maximum age is 80 years and minimum age is 0.42 years which is 5 months.- The graph for the Age is not a perfectly normal distribution and has a little skewness to the right.- Hence it is better to plot a boxplot for the Age variable<jupyter_code>sns.boxplot(df["Age"])<jupyter_output><empty_output><jupyter_text>We can see that majority of the People are between 20 and 40 years of Age and the people whose ages are greater than ~63 are considered as outliers<jupyter_code>#Lets check if people with the outlier ages survived.
outlier_ages = df[df["Age"]>63.0]
outlier_ages["Survived"].value_counts()
outlier_ages[outlier_ages["Survived"]==1][["Pclass", "Sex"]]<jupyter_output><empty_output><jupyter_text>Only 1 person from the outlier age group survived out of 13 and he was a male from 1st class.  Transformation 1 Age is continuous variable. We can convert it into a categorical variable by binning the values into three categories.- Children (0-17 yrs)- Adults (18-60)- Senior (>60)This way it will be better to analyse the data more clearly<jupyter_code>df.head()
def convertAge(x):
    if x>=0 and x<=17:
        return "Children";
    elif x>=18 and x<60:
        return "Adult";
    elif x>=60:
        return "Senior Citizen";
    
    
df["Age_Category"] = df["Age"].apply(convertAge)
df.head()

df = change_column_list(df, "Age_Category", 5)
df.head()
#Let's check, how many of the passengers from each of the three newly created age categories survived 
age = pd.DataFrame(df.groupby(["Age_Category", "Survived"])["Survived"].count().unstack().unstack().reset_index())
age.rename(columns={0:"Survival Count"}, inplace=True)
sns.factorplot("Age_Category", "Survival Count", hue="Survived", data=age, kind="bar");<jupyter_output><empty_output><jupyter_text>We can see that:- The non-survival count for adult age group ie. 18-60 is quite high due to the high death count of a number of males in that age group as we have seen before. - The count for Children i.e age group 0-17 seems to be higher. That means children had a high rate of survival amongst all the passengers.<jupyter_code># Let's also check which Pclass were the children who couldn't survive.
df[(df["Age_Category"]=="Children") & (df["Survived"]==0)]["Pclass"].value_counts()
# Let's check the SibSp feature now 
print(df["SibSp"].value_counts())

# and the Parch column
df["Parch"].value_counts()<jupyter_output><empty_output><jupyter_text>Transformation 2<jupyter_code># The SibSp and Parch columns represent whether the passenger had family memnbers accompanying them. 
# Let's add both columns into one called Family members.
df["Family_members"] = df['SibSp'] + df["Parch"]
df = change_column_list(df, "Family_members", 8)
df.head()
df["Family_members"].value_counts()
ff = pd.DataFrame(df.groupby(["Survived", "Family_members"])["Family_members"].agg("count").unstack().unstack().reset_index())
sns.factorplot("Survived", 0, hue="Family_members", data=ff, kind="bar");
plt.ylabel("Count of Survived/Non-Survived");<jupyter_output><empty_output><jupyter_text>- It can be seen that majortiy of the people who did not survive did not have any friends or family.- Also it can be seen that the non-survival rate for passengers having 4 and 5 members as friends or family, is higher than their survival rate.<jupyter_code># Let's check these passengers
fofi = df[(df["Family_members"]==4) | (df["Family_members"]==5)]
fofi[fofi["Survived"]==1]<jupyter_output><empty_output><jupyter_text>It can be seen that all those who survived were females from 1st and 2nd class<jupyter_code>fofi[(fofi["Pclass"]==1) | (fofi["Pclass"]==2)]<jupyter_output><empty_output><jupyter_text>And it can also be said that the first class males made way for women first so that they could survive<jupyter_code># Let's check the ticket column
df["Ticket"].nunique()<jupyter_output><empty_output><jupyter_text>Ticket is a categorical column with 681 categories. It makes sense to drop this column because it does not seem to provide much infromation.<jupyter_code>#Let's check the Fare column
print("Minimum Fare: ",df["Fare"].min())
print("Maximum Fare: ", df["Fare"].max())
sns.kdeplot(df["Fare"]);<jupyter_output>Minimum Fare:  0.0
Maximum Fare:  512.3292<jupyter_text>- The fare column is heavily skewed to the right. Which is expected because majority of the passengers were from the 3rd class and third class fares should be typically low and towards the right would have been the ticket fares of 2nd and 1st class passengers.- We will have to transform the Fare column using a scaling method  Step 5: Prepare the data for Machine Learning Algorithms  Transfromation 3<jupyter_code>#Lets make the PassengerID column the index
df.set_index("PassengerId", inplace=True)<jupyter_output><empty_output><jupyter_text>Transformation 4<jupyter_code>#Let's drop the unnecessary columns - Name, Ticket, Cabin
df = df.drop(["Name", "Ticket", "Cabin"], axis=1)
df.head()<jupyter_output><empty_output><jupyter_text>Transformation 5<jupyter_code># Let's drop the columns Age, SibSp and Parch which we have already converted
df = df.drop(["Age", "SibSp", "Parch"], axis=1)
df.head()<jupyter_output><empty_output><jupyter_text>Transformation 6<jupyter_code># The Age_Category has some None values which are basically NaN values from the original continuous column. Let us replace them with
# the most largely occuring value in Age_category ie. Adult
def fillAge(x):
    if x==None:
        return "Adult"
    else:
        return x
df["Age_Category"] = df["Age_Category"].apply(fillAge)

df.head()<jupyter_output><empty_output><jupyter_text>Transformation 7<jupyter_code># Replacing NaN values in Embarked with S - the most repeated value.
df['Embarked'].fillna("S", inplace=True)
df['Embarked'].isnull().sum()<jupyter_output><empty_output><jupyter_text>Transformation 8<jupyter_code># Applying Min-Max Scaling to the Fare variable
from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()
df[["Fare"]] = scaler.fit_transform(df[["Fare"]])<jupyter_output><empty_output><jupyter_text>Transformation 9<jupyter_code># Applying One-Hot Encoding on the categorical features
df = pd.get_dummies(df)
# Finally our data looks like this
df.head()
# Let us define a transfromation function that will apply all the transformations we did on train data to our test data
def transformation(df):
    df["Age_Category"] = df["Age"].apply(convertAge)
    df["Family_members"] = df['SibSp'] + df["Parch"]
    df.set_index("PassengerId", inplace=True)
    df = df.drop(["Name", "Ticket", "Cabin"], axis=1)
    df = df.drop(["Age", "SibSp", "Parch"], axis=1)
    df["Age_Category"] = df["Age_Category"].apply(fillAge)
    df['Embarked'].fillna("S", inplace=True)
    df[["Fare"]] = scaler.fit_transform(df[["Fare"]])
    df = pd.get_dummies(df)
    return df
# Loading the test data
df_test = pd.read_csv(test_dataset_path)

# Checking for Null Values before transforming.
df_test.isnull().sum()<jupyter_output><empty_output><jupyter_text>The Fare column has a null value that will not be replaced while tranformmation. Replacing this with the median value for Fare<jupyter_code>df_test["Fare"].fillna(df['Fare'].median(), inplace=True)
df_test = transformation(df_test)

df_test
# Loading the submission csv
submission = os.path.join(PROJECT_ROOT_DIR, ASSIGNMENT_ID, "titanic", "gender_submission.csv")
# Separating features and target
X = df.loc[:, df.columns!="Survived"]
y = df.loc[:, df.columns=="Survived"]
# Splitting the data into training and validation with an 80:20 ratio
from sklearn.model_selection import train_test_split

X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)<jupyter_output><empty_output><jupyter_text>Step 6: Select and train a model<jupyter_code># Testing KNN model

from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score
from sklearn.model_selection import StratifiedKFold, GridSearchCV, KFold

print("KNN model Results: ")

knn = KNeighborsClassifier(n_neighbors=3)
knn.fit(X_train, y_train)
y_pred_val_knn = knn.predict(X_val)
y_pred_test_knn = knn.predict(df_test)
print("On Validation data, the accuracy is: ",accuracy_score(y_val, y_pred_val_knn))
print("On Test data, the accuracy is: ",accuracy_score(submission["Survived"], y_pred_test_knn))
# Testing Logistic Regression
from sklearn.linear_model import LogisticRegression

print("Logistic Regression model Results: ")

lr = LogisticRegression()
lr.fit(X_train, y_train)

y_pred_val_lr = lr.predict(X_val)

y_pred_test_lr = lr.predict(df_test)

print("On Validation data, the accuracy is: ",accuracy_score(y_val, y_pred_val_lr))
print("On Test data, the accuracy is: ",accuracy_score(submission["Survived"], y_pred_test_lr))
from sklearn.svm import SVC

print("Support Vector Machine Classifier results: ")

svc = SVC()
svc.fit(X_train, y_train)

y_pred_val_svc = svc.predict(X_val)

y_pred_test_svc = svc.predict(df_test)

print("On Validation data, the accuracy is: ",accuracy_score(y_val, y_pred_val_svc))
print("On Test data, the accuracy is: ",accuracy_score(submission["Survived"], y_pred_test_svc))<jupyter_output>Support Vector Machine Classifier results: 
On Validation data, the accuracy is:  0.8044692737430168
On Test data, the accuracy is:  0.9832535885167464<jupyter_text>Step 7: Fine tune the model<jupyter_code># Let's fine tune the KNN model

print("KNN Fine-tuned model: ")

p1 = range(1, 10)
p2 = ['euclidean', 'manhattan', 'minkowski']
# define grid search
grid_search = GridSearchCV(estimator=knn, param_grid={"n_neighbors":p1,"metric":p2}, n_jobs=-1, cv=StratifiedKFold(n_splits=3, random_state=42), scoring='accuracy')
grid_result = grid_search.fit(X_train, y_train)

y_pred_val_knn_gs = grid_search.predict(X_val)
y_pred_test_knn_gs = grid_search.predict(df_test)

print("On Validation data, the accuracy is: ",accuracy_score(y_val, y_pred_val_knn_gs))
print("On Test data, the accuracy is: ",accuracy_score(submission["Survived"], y_pred_test_knn_gs))<jupyter_output>KNN Fine-tuned model: 
On Validation data, the accuracy is:  0.8156424581005587
On Test data, the accuracy is:  0.9952153110047847<jupyter_text>Accuracy of KNN model on test data has increased from 80% to 96% after finetuning the model.<jupyter_code># Let's fine tune the logistic regression model

print("Logistic Regression Fine-tuned model: ")

p1 = ['liblinear', 'newton-cg', 'lbfgs']
p2 = ["l2"]
p3 = [10, 1.0, 0.1, 0.01]
# define grid search
grid_search = GridSearchCV(estimator=lr, param_grid={"solver":p1,"penalty":p2,"C":p3}, n_jobs=-1, cv=StratifiedKFold(n_splits=3, random_state=42), scoring='accuracy')
grid_result = grid_search.fit(X_train, y_train)

y_pred_val_lr_gs = grid_search.predict(X_val)
y_pred_test_lr_gs = grid_search.predict(df_test)

print("On Validation data, the accuracy is: ",accuracy_score(y_val, y_pred_val_lr_gs))
print("On Test data, the accuracy is: ",accuracy_score(submission["Survived"], y_pred_test_lr_gs))<jupyter_output>Logistic Regression Fine-tuned model: 
On Validation data, the accuracy is:  0.7821229050279329
On Test data, the accuracy is:  0.8899521531100478<jupyter_text>GridSearchCV doesn't seem to made any difference in the prediction of Logitic regression since the accuracy has remained the same on the test data i.e 88%<jupyter_code># Let's fine tune the SVM model

print("SVM Fine-tuned model: ")

p1 = ['sigmoid', 'poly', 'rbf']
p2 = [10, 1.0, 0.1, 0.01]
# define grid search
grid_search = GridSearchCV(estimator=svc, param_grid={"kernel":p1, "C":p2}, n_jobs=-1, cv=StratifiedKFold(n_splits=3, random_state=42), scoring='accuracy')
grid_result = grid_search.fit(X_train, y_train)

y_pred_val_svc_gs = grid_search.predict(X_val)
y_pred_test_svc_gs = grid_search.predict(df_test)

print("On Validation data, the accuracy is: ",accuracy_score(y_val, y_pred_val_svc_gs))
print("On Test data, the accuracy is: ",accuracy_score(submission["Survived"], y_pred_test_svc_gs))<jupyter_output>SVM Fine-tuned model: 
On Validation data, the accuracy is:  0.8044692737430168
On Test data, the accuracy is:  0.9234449760765551
{'fraction_non_alphanumeric': 0.10858412228275242, 'fraction_numerical': 0.024170941979161156, 'mean_word_length': 5.895696571845368, 'pattern_counts': {'":': 8, '<': 126, '<?xml version=': 0, '>': 129, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '32', 'dir': 'jupyter-structured-clean-dedup', 'id': '645579', 'license': 'MIT', 'n_tokens_mistral': 6673, 'n_tokens_neox': 6236, 'n_words': 1957, 'path': 'Assignments/DSCI_633_Assignment_03.ipynb', 'repo_name': 'pranav-nair-rit/RIT-DSCI-633-FDS', 'size': '18912'}