<jupyter_start><jupyter_text>*from https://www.kaggle.com/c/titanic***Group project from the Purple Power Rangers (sg_prpl_power_rangers): <NAME> (@Elena), <NAME> (@Elena Kutanov), and <NAME> (@Eileen)**  Introduction to the ProjectIn our study group, we are all beginners when in comes to Deep Learning and none of us have entered in a kaggle competition before. The purpose of our study group was to find beginner friedly kaggle projects to get some hands-on experience with Deep Learning and also to get to know kaggle competitions. The Titanic competition is the classic first step when starting out at kaggle and a good intro to Machine Learning. That is why we chose it as our first project. At first each of us worked through a manual (*https://www.kaggle.com/sashr07/kaggle-titanic-tutorial*) which gives a good overview on how to prepare the data, train a model and submit the results to kaggle. The next step was to set-up our own model using **PyTorch** and trying to get the best predictions possible. Our journey with the Titanic project is documented in the following.   Following the manualAt first we will have a short summary of the manual, at least the points that we used as starting points for our own models.  Preparing the DataA lot of the data preparation process could be taken from the manual, but because we wanted to use PyTorch we had to make some changes.<jupyter_code>#loading the libraries and looking at the training and testing data
import pandas as pd
import torch as th
import matplotlib.pyplot as plt
import numpy as np

import warnings
warnings.simplefilter('ignore')

test = pd.read_csv("test.csv")
print(test.shape)

train = pd.read_csv("train.csv")
print(train.shape)
train.head()
test.head()<jupyter_output><empty_output><jupyter_text>train.csv and test.csv:Both training and testing data contain information about the passengers, but the information about survival is missing in the test data. The test data is for submission to kaggle. For model evaluation we will need to split our training data later on.   Which columns might be important to predict survival?From what we know about the Titanic disaster sex, age, and class were probably important factors to decide who survived. In the following we will look at these columns.<jupyter_code>sex_pivot = train.pivot_table(index="Sex",values="Survived")

sex_pivot.plot.bar()
plt.show()
pclass_pivot = train.pivot_table(index="Pclass",values="Survived")
pclass_pivot.plot.bar()
plt.show()
train['Age'].describe()
survived = train[train["Survived"] == 1]
died = train[train["Survived"] == 0]
survived["Age"].plot.hist(alpha=0.5,color='red',bins=50)
died["Age"].plot.hist(alpha=0.5,color='blue',bins=50)
plt.legend(['Survived','Died'])
plt.show()<jupyter_output><empty_output><jupyter_text>For the age column it is more convenient to work with categories. The following function will cut the age column into 6 age ranges plus a category for missing values.<jupyter_code>def process_age(df,cut_points,label_names):
    df["Age"] = df["Age"].fillna(-0.5)
    df["Age_categories"] = pd.cut(df["Age"],cut_points,labels=label_names)
    return df

cut_points = [-1,0, 5, 12, 18, 35, 60, 100]
label_names = ["Missing", 'Infant', "Child", 'Teenager', "Young Adult", 'Adult', 'Senior']

train = process_age(train,cut_points,label_names)
test = process_age(test,cut_points,label_names)

age_cat_pivot = train.pivot_table(index="Age_categories",values="Survived")
age_cat_pivot.plot.bar()
plt.show()<jupyter_output><empty_output><jupyter_text>Dummy columnsFor the model we need dummy columns representing sex, age categories, and class.<jupyter_code>def create_dummies(df,column_name):
    dummies = pd.get_dummies(df[column_name],prefix=column_name)
    df = pd.concat([df,dummies],axis=1)
    return df
train = create_dummies(train,"Pclass")
test = create_dummies(test,"Pclass")
train = create_dummies(train,"Sex")
test = create_dummies(test,"Sex")
train = create_dummies(train,"Age_categories")
test = create_dummies(test,"Age_categories")
train.head()
# The following columns will enter our model to predict survival onboard the Titanic:
columns = ['Pclass_1', 'Pclass_2', 'Pclass_3', 'Sex_female', 'Sex_male',
       'Age_categories_Missing','Age_categories_Infant',
       'Age_categories_Child', 'Age_categories_Teenager',
       'Age_categories_Young Adult', 'Age_categories_Adult',
       'Age_categories_Senior']
columns<jupyter_output><empty_output><jupyter_text>Splitting the Training DataTo have some validation for our model we need to split the training data into training and test (what was called _test_ before, will now be called _holdout_). To get reproducible results, we set a `random_state = 30`.<jupyter_code>holdout = test # from now on we will refer to this dataframe as the holdout data

from sklearn.model_selection import train_test_split

columns = ['Pclass_1', 'Pclass_2', 'Pclass_3', 'Sex_female', 'Sex_male',
       'Age_categories_Missing','Age_categories_Infant',
       'Age_categories_Child', 'Age_categories_Teenager',
       'Age_categories_Young Adult', 'Age_categories_Adult',
       'Age_categories_Senior']

all_X = train[columns]
all_y = train['Survived']

train_X, test_X, train_y, test_y = train_test_split(all_X, all_y, test_size=0.2, random_state = 30)
train_X.shape
all_X.shape<jupyter_output><empty_output><jupyter_text>The Model (from the Manual)We will now show the model from the manual, to have something to compare our results to.<jupyter_code>from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

# setting the seed for reproducibility:
#np.random.seed(30)

lr = LogisticRegression()
lr.fit(train_X, train_y)
predictions = lr.predict(test_X)
accuracy = accuracy_score(test_y, predictions)
accuracy *100.<jupyter_output><empty_output><jupyter_text>Accuracy:The logistic regression model from _sklearn_ gives us an **accuracy of 79.3%**.   Logistic Regression with PyTorchAs a first try we will build a logistic regression model just as in the manual but based on PyTorch  Data PreparationMostly we can use what we already have, but we decided to devide the age column more categories.<jupyter_code>def process_table(db):
    cut_points = [-1,0, 1, 4, 6, 12, 18, 28, 45, 60, 100]
    label_names = ["Missing", 'Infant', 'Baby', 'Toddler', "Child", 'Teenager', "Young Adult", 'Adult',
                   'Older Adult', 'Senior']
    # process age column
    db = process_age(db,cut_points,label_names)
    db = create_dummies(db, "Age_categories")
    #process sex column
    db = create_dummies(db, 'Sex')
    #process class column
    db = create_dummies(db,"Pclass")
    
    columns = ['Pclass_1', 'Pclass_2', 'Pclass_3', 'Sex_female', 'Sex_male',
               'Age_categories_Missing','Age_categories_Infant',
               'Age_categories_Baby', 'Age_categories_Toddler',
               'Age_categories_Child', 'Age_categories_Teenager',
               'Age_categories_Young Adult', 'Age_categories_Adult',
               'Age_categories_Older Adult','Age_categories_Senior']
    return db[columns]
#reload training data and create necessary dummy columns:
train = pd.read_csv("train.csv")
data = process_table(train)
labels = train['Survived']<jupyter_output><empty_output><jupyter_text>Splitting the training data into train and test dataJust as before, but because of the new age categories we need to do this again.<jupyter_code>train_X, test_X, train_y, test_y = train_test_split(data, labels, test_size=0.2, random_state = 30)<jupyter_output><empty_output><jupyter_text>Convert to torch tensorsOur input and label data needs to be converted to torch tensors.<jupyter_code>trainX = th.tensor(train_X.values)
trainX.shape
testX = th.tensor(test_X.values)
trainY = th.tensor(train_y.values)
testY = th.tensor(test_y.values)<jupyter_output><empty_output><jupyter_text>Define and Train the Model<jupyter_code>#import libraries
import torch.nn as nn
import torchvision.transforms as transforms
from torch.autograd import Variable

# setting the seed for reproducibility:
th.manual_seed(30)
# Create Logistic Regression Model
class LogisticRegressionModel(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(LogisticRegressionModel, self).__init__()
        
        self.linear = nn.Linear(input_dim, output_dim)
    
    def forward(self, x):
        out = self.linear(x)
        return out
    
# Instantiate Model Class
input_dim = 15 # number of input columns
output_dim = 2  # survived or not

# create logistic regression model
model = LogisticRegressionModel(input_dim, output_dim)

# Cross Entropy Loss  
error = nn.CrossEntropyLoss()

# SGD Optimizer 
learning_rate = 0.001
optimizer = th.optim.SGD(model.parameters(), lr=learning_rate)
# Traning the Model
batch_size = 4
batches = 178
num_epochs = 25

loss_list = []
accuracy_list = []
for epoch in range(num_epochs):
    for i in range(batches):
        
        # Clear gradients
        optimizer.zero_grad()
        
        # Forward propagation
        start = i * batch_size
        end = start + batch_size
        outputs = model(trainX[start:end].float())
        
        # Calculate softmax and cross entropy loss
        loss = error(outputs, trainY[start:end].long())
        
        # Calculate gradients
        loss.backward()
        
        # Update parameters
        optimizer.step()
        
        # Prediction
        with th.no_grad(): 
            model.eval()
            # Predict test dataset
            outputs = model(testX.float())
                
            # Get predictions from the maximum value
            outputs = th.max(outputs, 1)[1]
            accuracy = th.mean((outputs == testY.long()).float())
        model.train()
            
        # store loss
        loss_list.append(loss.data)
        
    print('epoch {}: Accuracy {:.2f}'.format(epoch+1,accuracy * 100.0))
    accuracy_list.append(accuracy)
epochs = range(num_epochs)
plt.plot(epochs,accuracy_list)
plt.legend(['final accuracy = {:.2f}%'.format(np.mean(accuracy_list[-10:-1]) * 100.0)])
plt.xlabel('epochs');<jupyter_output><empty_output><jupyter_text>With this model we get an **accuracy of 79.3%**, so just as in the model from the manual.   Deep Learning ModelAs a next step we employ a deep learning model with 2 fully connected layers.  Defining and Training the Model<jupyter_code>import torch.nn.functional as F

# setting the seed for reproducibility:
th.manual_seed(30)

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(15, 45)
        self.fc2 = nn.Linear(45, 1)
        self.dropout = nn.Dropout(p=0.2)
        
    def forward(self, x):
        x = F.sigmoid(self.fc1(x))
        x = self.dropout(x)
        x = self.fc2(x)
        return x
epoches = 500
batch_size = 89
batches = 8
model = Net()
criterion = nn.BCEWithLogitsLoss()
optimizer = th.optim.Adam(model.parameters(), lr=0.001)  

accuracy_list = []

for epoch in range(epoches):
    for i in range(batches):
        optimizer.zero_grad()
        start = i * batch_size
        end = start + batch_size
        output = model(trainX[start:end].float())
        loss = criterion(output, trainY[start:end].float().view(-1, 1))
        loss.backward()
        optimizer.step()
    with th.no_grad():
        model.eval()
        output = th.exp(model(testX.float()))
        output = (output > 0.5).long()
        accuracy = th.mean((output == testY.view(-1, 1).long()).float())
        accuracy_list.append(accuracy)
    if (epoch % 50 == 0): 
        print('epoch {}: Accuracy {:.2f}'.format(epoch,accuracy * 100.0))
    model.train()

eps = range(epoches)
plt.plot(eps,accuracy_list)
plt.legend(['final accuracy = {:.2f}%'.format(np.mean(accuracy_list[-10:-1]) * 100.0)])
plt.xlabel('epochs');<jupyter_output><empty_output><jupyter_text>Accuracy:This model yields an **accuracy score of 75.4%**, so slightly less than the logistic regression models. We will try one more model, where we adjust the age categories back to the same state as in the manual (just 6).   Deep Learning Model With Less Age Categories  Data Preparation As before, but with less age categories:<jupyter_code>train = pd.read_csv('train.csv')
def process_age(df,cut_points,label_names):
    df["Age"] = df["Age"].fillna(-0.5)
    df["Age_categories"] = pd.cut(df["Age"],cut_points,labels=label_names)
    return df

cut_points = [-1,0, 5, 12, 18, 35, 60, 100]
label_names = ["Missing", 'Infant', "Child", 'Teenager', "Young Adult", 'Adult', 'Senior']

train = process_age(train, cut_points, label_names)
def create_dummies(df,column_name):
    dummies = pd.get_dummies(df[column_name],prefix=column_name)
    df = pd.concat([df,dummies],axis=1)
    return df

def prepare_data(data):
    data = create_dummies(data,"Pclass")
    data = create_dummies(data,"Sex")
    data = create_dummies(data,"Age_categories")
    
    return data
columns = ['Pclass_1', 'Pclass_2', 'Pclass_3', 'Sex_female', 'Sex_male',
       'Age_categories_Missing','Age_categories_Infant',
       'Age_categories_Child', 'Age_categories_Teenager',
       'Age_categories_Young Adult', 'Age_categories_Adult',
       'Age_categories_Senior']
train = prepare_data(train)
print('train.shape:', train.shape)<jupyter_output>train.shape: (891, 25)<jupyter_text>Splitting the Data in to Training and Testing Data<jupyter_code>all_X = train[columns]
all_y = train['Survived']

train_X, test_X, train_Y, test_Y = train_test_split(all_X, all_y, test_size=0.2, random_state=30)<jupyter_output><empty_output><jupyter_text>Convert to PyTorch<jupyter_code>print('---> train_X:', train_X.shape)
print('---> train_Y:', train_Y.shape)
print('---> test_X:', test_X.shape)
print('---> test_Y:', test_Y.shape)

train_X_t = th.tensor(train_X.values)
train_Y_t = th.tensor(train_Y.values)

test_X_t  = th.tensor(test_X.values)
test_Y_t  = th.tensor(test_Y.values)<jupyter_output>---> train_X: (712, 12)
---> train_Y: (712,)
---> test_X: (179, 12)
---> test_Y: (179,)<jupyter_text>Define and Train the Model<jupyter_code># *********** Create the model **********
#from torch import nn
from collections import OrderedDict

# setting the seed for reproducibility:
th.manual_seed(30)

model = nn.Sequential(OrderedDict([
    ('fc1',  nn.Linear(12, 4)),
    ('relu', nn.ReLU()),
    ('drop', nn.Dropout(p=0.2)),
    ('fc2',  nn.Linear(4, 2)),
    ('output', nn.Softmax(dim=1))
]))
def model_learning(obj_model, obj_opt, obj_data, obj_target):
    obj_opt.zero_grad()
    
    obj_pred = obj_model(obj_data.float())
    obj_loss = criterion(obj_pred, obj_target.long())
    obj_loss.backward()
    
    obj_opt.step()

    return obj_loss
epochs = 200
train_sets_num = 89
train_set_size = 8

# setting the seed for reproducibility:
th.manual_seed(30)

loss = th.FloatTensor
criterion = nn.CrossEntropyLoss()
opt = th.optim.Adam(model.parameters(), lr=0.01)

accuracy_list = []

for e in range(epochs):
    running_loss = 0
    for ts_i in range(0, train_sets_num, train_set_size):
        running_loss += model_learning(model, opt,
                                      train_X_t[ts_i: ts_i+train_set_size],
                                      train_Y_t[ts_i: ts_i+train_set_size ]).item()
    else: 
        with th.no_grad():
            model.eval()
            test_loss = 0
            accuracy  = 0

            pred = model(test_X_t.float())
            loss = criterion(pred, test_Y_t.long())
            test_loss += loss.item()

            output = th.max(pred, 1)[1]
            accuracy = th.mean((output == test_Y_t.long()).float())
            accuracy_list.append(accuracy)

    if (e % 10 == 0): 
        print('epoch {}: Accuracy {:.2f}'.format(e,accuracy * 100.0))
            
        model.train()
eps = range(epochs)
plt.plot(eps,accuracy_list)
plt.legend(['final accuracy = {:.2f}%'.format(np.mean(accuracy_list[-10:-1]) * 100.0)])
plt.xlabel('epochs');<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11831869510664994, 'fraction_numerical': 0.0157465495608532, 'mean_word_length': 4.115853658536586, 'pattern_counts': {'":': 0, '<': 62, '<?xml version=': 0, '>': 71, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '433593', 'license': 'Apache-2.0', 'n_tokens_mistral': 5516, 'n_tokens_neox': 5166, 'n_words': 1388, 'path': 'Titanic/Titanic_report.ipynb', 'repo_name': 'bharatnishant/UdacityOpenSource', 'size': '15963'}