<jupyter_start><jupyter_text>Titanic Logistic Regression<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import tensorflow as tf<jupyter_output><empty_output><jupyter_text>![titanic-disaster](./titanic-disaster.jpg)<jupyter_code>train = pd.read_csv('train.csv')
test = pd.read_csv('test.csv')
train.head(10)<jupyter_output><empty_output><jupyter_text>Data Dictionary   - Survival: 0 = no, 1 = yes   - Pclass: 1 = 1st, 2 = 2nd, 3 = 3rd   - Sibsp: of siblings / spouses aboard the Titanic   - Parch: of parents / children aboard the Titanic   - Embarked: C = Cherbourg, Q = Queenstown, S = Southampton Variable Notes   - pclass: A proxy for socio-economic status (SES)        - 1st = Upper        - 2nd = Middle        - 3rd = Lower   - age: Age is fractional if less than 1. If the age is estimated, is it in the form of xx.5   - sibsp: The dataset defines family relations in this way...   - Sibling = brother, sister, stepbrother, stepsister   - Spouse = husband, wife (mistresses and fiancés were ignored)   - parch: The dataset defines family relations in this way...   - Parent = mother, father   - Child = daughter, son, stepdaughter, stepson   - Some children travelled only with a nanny, therefore parch=0 for them.<jupyter_code>train.shape
train.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 12 columns):
PassengerId    891 non-null int64
Survived       891 non-null int64
Pclass         891 non-null int64
Name           891 non-null object
Sex            891 non-null object
Age            714 non-null float64
SibSp          891 non-null int64
Parch          891 non-null int64
Ticket         891 non-null object
Fare           891 non-null float64
Cabin          204 non-null object
Embarked       889 non-null object
dtypes: float64(2), int64(5), object(5)
memory usage: 83.6+ KB<jupyter_text>Check missing datas.. *Age, Cabin, Embarked*especially datas of *Cabin* were missing a lot<jupyter_code>train.isna().sum()<jupyter_output><empty_output><jupyter_text>We can check how much datas are lost per column using *pd.isna()* or *pd.isnull()*There were lost 177 datas in column of *Age*, 687 datas  in column of *Cabin*, 2 datas in column of *Embarked*<jupyter_code>def check(column):
    survived = train[train['Survived'] == 1][column].value_counts()
    dead = train[train['Survived'] == 0][column].value_counts()
    df = pd.DataFrame([survived,dead])
    df.index = ['Survived', 'Dead']
    df.plot(kind = 'bar', stacked =True,figsize = (10,5))
check('Sex')<jupyter_output><empty_output><jupyter_text>This chart shows that **women** survived more than **men.**<jupyter_code>check('Pclass')<jupyter_output><empty_output><jupyter_text>This chart shows that the lower the **'pclass'**, the more deathsAnd on the contrary, more people of higher pclass survived.<jupyter_code>def count_pclass():
    fir = np.asarray(train['Pclass'] == 1).sum()
    sec = np.asarray(train['Pclass'] == 2).sum()
    thir = np.asarray(train['Pclass'] == 3).sum()
    df = pd.DataFrame([fir, sec, thir])
    df.index = ['First','Second','Third']
    df.plot(kind = 'bar', figsize = (10,5))
    
count_pclass()<jupyter_output><empty_output><jupyter_text>but you should know, of all passengers, 3 pclass is the most frequent<jupyter_code>check('Parch')<jupyter_output><empty_output><jupyter_text>This chart shows that passengers who aboard a ship alone without their families died more than other passengers.<jupyter_code>check('Embarked')<jupyter_output><empty_output><jupyter_text>Passengers aboard **'Q'** died more than other passengershmm, let me guess more of people were aboard from **'Q'** check it out<jupyter_code>def count():
    S = train[train['Embarked'] == 'S']['Pclass'].value_counts()
    C = train[train['Embarked'] == 'C']['Pclass'].value_counts()
    Q = train[train['Embarked'] == 'Q']['Pclass'].value_counts()
    df = pd.DataFrame([S, C, Q])
    df.index = ['S','C','Q']
    df.plot(kind = 'bar', figsize = (10,5))
    
count()<jupyter_output><empty_output><jupyter_text>i got it! , most of people were aboard from **'Q'** and a lot of people aboard from **'Q'** who has class 3This is the Titanic route. We know where Q, S, C are![titanic-way](./titanic-way.jpg)  Feature engineeringFeature engineering is the process of using domain knowledge of data.Generates features (feature vectors) that activate machine learning algorithms.First, combining train and test datas, and we should change name to number<jupyter_code>train_test_data = [train, test]
for data in train_test_data:
    data['Title'] = data['Name'].str.extract(' ([A-Za-z]+)\.', expand=False)

print('In Train data: \n', train['Title'].value_counts())
print('\nIn Test data: \n', test['Title'].value_counts())
train['Title'][:10]<jupyter_output><empty_output><jupyter_text>We gonna change Name to Number like this**Map**Mr : 0Miss : 1Mrs : 2Others : 3<jupyter_code>title_mapping = {"Mr": 0, "Miss": 1, "Mrs": 2, 
                 "Master": 3, "Dr": 3, "Rev": 3, "Col": 3, "Major": 3, "Mlle": 3,"Countess": 3,
                 "Ms": 3, "Lady": 3, "Jonkheer": 3, "Don": 3, "Dona" : 3, "Mme": 3,"Capt": 3,"Sir": 3 }
for data in train_test_data:
    data['Title'] = data['Title'].map(title_mapping)
train.head()
check('Title')
train.drop('Name', axis = 1, inplace = True)<jupyter_output><empty_output><jupyter_text>We gonna mapping **embarked and sex ** too<jupyter_code>embarked_mapping = {"S": 0, "C": 1, "Q": 2 }
sex_mapping = {"male": 0, "female": 1,}

for data in train_test_data:
    data['Embarked'] = data['Embarked'].fillna('S')
    data['Embarked'] = data['Embarked'].map(embarked_mapping)
    data['Sex'] = data['Sex'].map(sex_mapping)
train.head()<jupyter_output><empty_output><jupyter_text>Fill missing datafirst, we know that some of age data is missing, we going to fill missing age data with median age for each title (Mr, Mrs, Miss, etc...)<jupyter_code>train["Age"].fillna(train.groupby("Title")["Age"].transform("median"), inplace=True)
test["Age"].fillna(test.groupby("Title")["Age"].transform("median"), inplace=True)
train.head(30)
train.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 12 columns):
PassengerId    891 non-null int64
Survived       891 non-null int64
Pclass         891 non-null int64
Sex            891 non-null int64
Age            891 non-null float64
SibSp          891 non-null int64
Parch          891 non-null int64
Ticket         891 non-null object
Fare           891 non-null float64
Cabin          204 non-null object
Embarked       891 non-null int64
Title          891 non-null int64
dtypes: float64(2), int64(8), object(2)
memory usage: 83.6+ KB<jupyter_text>we also going to mapping to age map:child: 0  young: 1  adult: 2  mid-age: 3  senior: 4<jupyter_code>for dataset in train_test_data:
    dataset.loc[dataset['Age'] <= 16, 'Age'] = 0,
    dataset.loc[(dataset['Age'] > 16) & (dataset['Age'] <= 26), 'Age'] = 1,
    dataset.loc[(dataset['Age'] > 26) & (dataset['Age'] <= 36), 'Age'] = 2,
    dataset.loc[(dataset['Age'] > 36) & (dataset['Age'] <= 62), 'Age'] = 3,
    dataset.loc[ dataset['Age'] > 62, 'Age'] = 4    
train.head()<jupyter_output><empty_output><jupyter_text>we going to fill missing fare data with median fare for each Pclass<jupyter_code>train['Fare'].fillna(train.groupby('Pclass')['Fare'].transform('median'), inplace=True)
test['Fare'].fillna(test.groupby('Pclass')['Fare'].transform('median'), inplace=True)
train.head()<jupyter_output><empty_output><jupyter_text>ok, so we have to fill missing cabin data too<jupyter_code>train['Cabin'].value_counts()
for dataset in train_test_data:
    dataset['Cabin'] = dataset['Cabin'].str[:1]
p1 = train[train['Pclass'] == 1]['Cabin'].value_counts()
p2 = train[train['Pclass'] == 2]['Cabin'].value_counts()
p3 = train[train['Pclass'] == 3]['Cabin'].value_counts()
df = pd.DataFrame([p1, p2, p3])
df.index = ['1st class','2nd class', '3rd class']
df.plot(kind='bar',stacked=True, figsize=(10,5))
cabin_mapping = {"A": 0, "B": 0.4, "C": 0.8, "D": 1.2, "E": 1.6, "F": 2, "G": 2.4, "T": 2.8}
for dataset in train_test_data:
    dataset['Cabin'] = dataset['Cabin'].map(cabin_mapping)<jupyter_output><empty_output><jupyter_text>Fill missing cabin data with median value for each pclass<jupyter_code>train["Cabin"].fillna(train.groupby("Pclass")["Cabin"].transform("median"), inplace=True)
test["Cabin"].fillna(test.groupby("Pclass")["Cabin"].transform("median"), inplace=True)<jupyter_output><empty_output><jupyter_text>make family size data using SibSp and Parch<jupyter_code>train["FamilySize"] = train["SibSp"] + train["Parch"] + 1
test["FamilySize"] = test["SibSp"] + test["Parch"] + 1
family_mapping = {1: 0, 2: 0.4, 3: 0.8, 4: 1.2, 5: 1.6, 6: 2, 7: 2.4, 8: 2.8, 9: 3.2, 10: 3.6, 11: 4}
for dataset in train_test_data:
    dataset['FamilySize'] = dataset['FamilySize'].map(family_mapping)
train.head()<jupyter_output><empty_output><jupyter_text>Modelingi'm gonna using simple linear regression with batch normalization to classification<jupyter_code>features_drop = ['Ticket', 'SibSp', 'Parch']
train = train.drop(features_drop, axis=1)
test = test.drop(features_drop, axis=1)
train = train.drop(['PassengerId'], axis=1)
Xdata = train.drop('Survived', axis=1)
Ydata = train['Survived']

Xdata.shape, Ydata.shape
Xdata = np.array(Xdata)
Ydata = np.array(Ydata)
Ydata = Ydata.reshape(891,1)
Ydata.shape<jupyter_output><empty_output><jupyter_text>Implement custom Batch Generator if you wanna see more about my batch generator https://github.com/che9992/BatchGenerator<jupyter_code>class BatchGenerator():
    where = 0
    '''
    usage
    
    var = BatchGenerator(xdata, ydata, batch_size = 100)
    var.x 
    var.y
    var.next()
    
    '''
    
    def __init__(self, x, y, batch_size, one_hot = False, nb_classes = 0):
        self.nb_classes = nb_classes
        self.one_hot = one_hot
        self.x_ = x
        self.y_ = y
        self.batch_size = batch_size
        
        self.total_batch = int(len(x) / batch_size)
        self.x = self.x_[:batch_size,:]
        self.y = self.y_[:batch_size,:]
        self.where = batch_size
        
        if self.one_hot :
            self.set_one_hot()

    def next_batch(self):
        if self.where + self.batch_size > len(self.x_) :
            self.where = 0
            
        self.x = self.x_[self.where:self.where+self.batch_size,:]
        self.y = self.y_[self.where:self.where+self.batch_size,:]
        self.where += self.batch_size
        
        if self.one_hot:
            self.set_one_hot()
        
    def set_one_hot(self):
        self.y = np.int32(self.y)
        one_hot = np.array(self.y).reshape(-1)
        self.y = np.eye(self.nb_classes)[one_hot].astype(np.int)<jupyter_output><empty_output><jupyter_text>Using Batch nomalization before ReLu if you wanna see more about BN check it out https://github.com/che9992/Batch_Normalization<jupyter_code>def TO_FC_BN(X, size, phase, scope):
    with tf.variable_scope(scope):
        fc1 = tf.contrib.layers.fully_connected(X, size, activation_fn=None, scope='fully_connected', reuse=tf.AUTO_REUSE)
        fc2 = tf.contrib.layers.batch_norm(fc1, center = True, scale = True, is_training= phase, scope='bn', reuse=tf.AUTO_REUSE)
        return tf.nn.relu(fc2, name='relu')
    
def FC_2_NC(X, nb_classes, scope):
    return tf.contrib.layers.fully_connected(X, nb_classes, activation_fn=None, scope=scope, reuse=tf.AUTO_REUSE)    
tf.reset_default_graph() 
nb_classes = 2

phase = tf.placeholder(dtype=tf.bool, name = 'phase')
X = tf.placeholder(shape=[None, 8], dtype=tf.float32, name='X')
Y = tf.placeholder(shape=[None, nb_classes], dtype=tf.float32, name='Y')

FC1 = TO_FC_BN(X, 128, phase=phase, scope='layer1')
FC2 = TO_FC_BN(FC1, 256, phase=phase, scope='layer2')
FC3 = TO_FC_BN(FC2, 512, phase=phase, scope='layer3')
FC4 = TO_FC_BN(FC3, 1024, phase=phase, scope='layer4')
FC5 = TO_FC_BN(FC4, 1024, phase=phase, scope='layer5')
FC6 = TO_FC_BN(FC5, 512, phase=phase, scope='layer6')
FC7 = TO_FC_BN(FC6, 512, phase=phase, scope='layer7')
FC8 = TO_FC_BN(FC7, 256, phase=phase, scope='layer8')
logits = FC_2_NC(FC8, nb_classes, scope='logits')


with tf.name_scope('accuracy'):
    correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(Y, 1))
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

with tf.name_scope('cost'):
     cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(logits = logits, labels = Y))
update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies(update_ops):
    optimizer = tf.train.AdamOptimizer(1e-4).minimize(cost)

batch = BatchGenerator(Xdata,Ydata,batch_size=100,nb_classes=2,one_hot=True)
check = BatchGenerator(Xdata,Ydata,batch_size=len(Xdata),nb_classes=2,one_hot=True)
sess = tf.Session()
sess.run(tf.global_variables_initializer())

def train(epochs = 20, batch_size = 200):
 
    for epoch in range(epochs):
        avg_cost = 0
        
        for i in range(batch.total_batch):
            c, _ = sess.run([cost, optimizer], feed_dict= {'X:0': batch.x, 'Y:0': batch.y, 'phase:0': True})
            avg_cost += c / batch.total_batch
            batch.next_batch()
            
        if epoch % 100 == 0:
            print("Epoch:", '%04d,' % (epoch), 'cost = ', '{:.9f}'.format(avg_cost))
            train_acc = sess.run(accuracy,feed_dict={'X:0': check.x, 'Y:0': check.y, 'phase:0': True})
            print('Train Accuracy: {:.2f}% '.format(train_acc * 100))

train(epochs=500, batch_size=400)
test = test.drop('Name',axis=1)
test_1 = test.drop('PassengerId',axis=1)
def tesing():
    t = sess.run(tf.argmax(logits, 1), feed_dict={'X:0': test_1, 'phase:0': False})
    return t
result = tesing()
submission = pd.DataFrame({
        "PassengerId": test['PassengerId'],
        "Survived": result
    })
submission.to_csv('submission.csv', index=False)
submission.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15394501963584434, 'fraction_numerical': 0.03141735094609068, 'mean_word_length': 4.319407519939233, 'pattern_counts': {'":': 32, '<': 93, '<?xml version=': 0, '>': 94, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '416025', 'license': 'MIT', 'n_tokens_mistral': 5339, 'n_tokens_neox': 4945, 'n_words': 1181, 'path': 'main.ipynb', 'repo_name': 'che9992/Kaggle_Titanic', 'size': '14005'}