<jupyter_start><jupyter_text>The Passenger ID and the Ticket ID are not needed:<jupyter_code>raw_df.drop("PassengerId", axis=1, inplace=True)
raw_df.drop("Ticket", axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>Preprocessing: Notice that we won't be using libraries (e.g. sklearn.preprocessing) for label encoding (except for Gender), but instead, we'll encode according to "Survival" chance (a feature or value with a higher survival chance will get a higher encoding number)<jupyter_code>from sklearn.preprocessing import LabelBinarizer
from sklearn.preprocessing import StandardScaler, RobustScaler, MinMaxScaler

scaler = StandardScaler()
pre_df = raw_df.copy()<jupyter_output><empty_output><jupyter_text>Check NaN values:<jupyter_code>raw_df.isnull().sum()<jupyter_output><empty_output><jupyter_text>Gender: Survival chance and Encoding Check Gender importance in survival:<jupyter_code>fig = plt.figure(figsize=(10, 4))

fig.add_subplot(121)
raw_df["Survived"][raw_df["Sex"] == "male"].value_counts().plot(kind="pie", title="Males", legend=True, colors=['c', 'y'])

fig.add_subplot(122)
raw_df["Survived"][raw_df["Sex"] == "female"].value_counts().plot(kind="pie", title="Females", legend=True, colors=['c', 'y']);
from sklearn.preprocessing import LabelBinarizer

pre_df["Sex"] = LabelBinarizer().fit_transform(raw_df["Sex"])<jupyter_output><empty_output><jupyter_text>Name / Title : Survival chance according to Title/!\ Encoding is done later<jupyter_code>pre_df["Name"] = raw_df["Name"].map(lambda x: x.split(',')[1].split('.')[0].strip())
titles = pre_df["Name"].unique()
print("# of titles:", len(titles))
print(titles)
fig = plt.figure(figsize=(15,6))

for title_i in range(len(titles)):
    fig.add_subplot(3, 6, title_i + 1)
    plt.title('Title : {}'.format(titles[title_i]))
    pre_df["Survived"][pre_df["Name"] == titles[title_i]].value_counts().plot(kind="pie", colors=['c', 'y'])<jupyter_output><empty_output><jupyter_text>Age: NaN fill and NormalizingWe'll fill NaNs with the median with regards to Title ('Mr', 'Master', ...)<jupyter_code>pre_df["Age"].fillna(-1, inplace=True)

medians = {}
for title_i in range(len(titles)):
    median = pre_df["Age"][(pre_df["Age"] != -1) & (pre_df["Name"] == titles[title_i])].median()
    medians[titles[title_i]] = median
    
for idx, row in pre_df.iterrows():
    if row["Age"] == -1:
        pre_df.loc[idx, "Age"] = medians[row["Name"]]
pre_df["Age"] = scaler.fit_transform(pre_df[["Age"]])<jupyter_output><empty_output><jupyter_text>Name / Title : (Next) Encoding and Normalizing<jupyter_code>replacement = {
    'Don': 0,
    'Rev': 0,
    'Jonkheer': 0,
    'Capt': 0,
    'Mr': 1,
    'Dr': 2,
    'Col': 3,
    'Major': 3,
    'Master': 4,
    'Miss': 5,
    'Mrs': 6,
    'Mme': 7,
    'Ms': 7,
    'Mlle': 7,
    'Sir': 7,
    'Lady': 7,
    'the Countess': 7
}

pre_df["Name"] = pre_df["Name"].apply(lambda x: replacement.get(x))
pre_df["Name"] = scaler.fit_transform(pre_df[["Name"]])<jupyter_output><empty_output><jupyter_text>Cabin: NaN fill, Survival chance and EncodingNotice that we'll fill NaNs with median with regards to passenger class "Pclass"<jupyter_code>pre_df["Cabin"].fillna('U', inplace=True)

pre_df["Cabin"] = pre_df["Cabin"].apply(lambda x: x[0])

cabins = pre_df["Cabin"].unique()
print(cabins)
fig = plt.figure(figsize=(15, 6))

for cab_i in range(len(cabins)):
    fig.add_subplot(2, 5, cab_i + 1)
    plt.title("Cabin: {}".format(cabins[cab_i]))
    pre_df["Survived"][pre_df["Cabin"] == cabins[cab_i]].value_counts().plot(kind="pie", colors=['c', 'y'])
#pre_df["Cabin"] = LabelEncoder().fit_transform(pre_df["Cabin"])

replacement = {
    'T': 0,
    'U': 1,
    'A': 2,
    'G': 3,
    'C': 4,
    'F': 5,
    'B': 6,
    'E': 7,
    'D': 8   
}

pre_df["Cabin"] = pre_df["Cabin"].apply(lambda x: replacement.get(x))
"""
classes = pre_df["Pclass"].unique()

medians = {}
for clss_i in range(len(classes)):
    median = pre_df["Cabin"][(pre_df["Cabin"] != 1) & (pre_df["Pclass"] == classes[clss_i])].median()
    medians[classes[clss_i]] = median
    
print(medians)

for idx, row in pre_df.iterrows():
    if row["Age"] == -1:
        pre_df.loc[idx, "Age"] = medians[row["Name"]]
"""
pre_df[["Cabin"]] = scaler.fit_transform(pre_df[["Cabin"]])<jupyter_output><empty_output><jupyter_text>Embarkment port: Chance of survival, NaN fill, and One Hot Encoding<jupyter_code>pre_df["Embarked"].value_counts()<jupyter_output><empty_output><jupyter_text>We see the majority embarked from 'S' (Southampton), so :<jupyter_code>pre_df["Embarked"].fillna('S', inplace=True)
emb_ports = pre_df["Embarked"].unique()

fig = plt.figure(figsize=(9, 3))
for emb_i in range(len(emb_ports)):
    fig.add_subplot(1, 3, emb_i + 1)
    plt.title("Embarked: {}".format(emb_ports[emb_i]))
    pre_df["Survived"][pre_df["Embarked"] == emb_ports[emb_i]].value_counts().plot(kind="pie", colors=['c', 'y'])
replacement = {
    'S': 0,
    'Q': 1,
    'C': 2
}

pre_df["Embarked"] = pre_df["Embarked"].apply(lambda x: replacement.get(x))
pre_df["Embarked"] = scaler.fit_transform(pre_df[["Embarked"]])<jupyter_output><empty_output><jupyter_text>Normalizing the rest:<jupyter_code>pre_df["Fare"] = scaler.fit_transform(pre_df[["Fare"]])
pre_df["SibSp"] = scaler.fit_transform(pre_df[["SibSp"]])
pre_df["Parch"] = scaler.fit_transform(pre_df[["Parch"]])
pre_df["Pclass"] = scaler.fit_transform(pre_df[["Pclass"]])
pre_df["Sex"] = scaler.fit_transform(pre_df[["Sex"]])<jupyter_output><empty_output><jupyter_text>Saving to CSV:<jupyter_code>pre_df.to_csv("./data/preproc_train.csv", header=True, index=False)<jupyter_output><empty_output><jupyter_text>Training Day:<jupyter_code>from sklearn.model_selection import train_test_split

x_train, x_test, y_train, y_test = train_test_split(
    pre_df.drop("Survived", axis=1),
    pre_df["Survived"],
    test_size=0.2
)<jupyter_output><empty_output><jupyter_text>Random Forest:<jupyter_code>from sklearn.ensemble import RandomForestClassifier

forest = RandomForestClassifier(n_jobs=-1, n_estimators=100, random_state=123)

forest.fit(x_train, y_train)
forest_train_score = forest.score(x_train, y_train)
print(forest_train_score)
forest_test_score = forest.score(x_test, y_test)
print(forest_test_score)<jupyter_output>0.826815642458<jupyter_text>TensorFlow NN:<jupyter_code>import tensorflow as tf
tf.logging.set_verbosity(tf.logging.INFO)
def get_batch(xs, ys, batch_size=50):
    
    assert(xs.shape[0] == ys.shape[0])

    idxs = np.random.permutation(range(xs.shape[0]))
    
    n_batches = len(xs) // batch_size
    
    for batch_i in range(n_batches):
        
        """
        batch_x = xs[batch_i * batch_size : (batch_i + 1) * batch_size]
        batch_y = ys[batch_i * batch_size : (batch_i + 1) * batch_size]
        """
        
        idxs_i = idxs[batch_i * batch_size : (batch_i + 1) * batch_size]
        
        batch_x = xs.iloc[idxs_i]
        batch_y = ys.iloc[idxs_i]
        yield batch_x, batch_y
def build_nn(n_inputs, n_outputs, activation=tf.nn.relu, n_hidden_layers=1,
             n_hidden_units=10, optimizer=tf.train.AdamOptimizer(learning_rate=0.01),
             scope=None):

    tf.reset_default_graph()

    with tf.variable_scope(scope or "neural_net", ):
        X = tf.placeholder(dtype=tf.float32, shape=[None, n_inputs])
        Y = tf.placeholder(dtype=tf.float32, shape=[None, n_outputs])

        Y_feedforward = X
        for _ in range(n_hidden_layers):
            Y_feedforward = tf.layers.dense(
                inputs=Y_feedforward,
                units=n_hidden_units,
                activation=activation,
                kernel_initializer=tf.contrib.layers.xavier_initializer()
            )

        # Output layer:
        logits = tf.layers.dense(inputs=Y_feedforward, units=1, activation=None)
        cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(labels=Y, logits=logits)
        cost = tf.reduce_mean(cross_entropy)
        optimizer = optimizer.minimize(cost)
        Y_pred = tf.round(tf.sigmoid(logits))

        return X, Y, Y_pred, cost, optimizer
def predict(xs, Y_pred, sess):
    with sess.as_default():
        return Y_pred.eval(
            session=sess,
            feed_dict={
                X: xs
            }
        )

def accuracy(y_pred, y_target):
    return np.mean(np.equal(y_pred, y_target).astype(np.float32))
def train(xs, ys, Y_pred, optimizer, loss, sess, epochs=1000, batch_size=50):
    
    with sess.as_default():
        
        costs = []
        accuracies = []
        n_batches = len(xs) // batch_size
        
        ys = ys.values.reshape(-1, 1)
        
        sess.run(tf.global_variables_initializer())
        
        for iter_i in range(1, epochs + 1):
            
            for batch_x, batch_y in get_batch(xs, pd.DataFrame(ys), batch_size):
                
                train_cost = sess.run(
                    fetches=[loss, optimizer],
                    feed_dict={
                        X: batch_x,
                        Y: batch_y
                    }
                )[0]
               
            
            if iter_i % 10 == 0:
                costs.append(train_cost)
                
                acc = accuracy(
                    y_pred=predict(xs=xs, Y_pred=Y_pred, sess=sess),
                    y_target=ys
                )
                accuracies.append(acc)
                
                print("Epoch: {}/{}".format(iter_i, epochs))
                print('\t', "Train Cost: {}".format(train_cost))
                print('\t', "Train Acc: {}".format(acc))
                
        return Y_pred
X, Y, Y_pred, cost, optim = build_nn(
    n_inputs=x_train.shape[1],
    n_outputs=1,
    activation=tf.nn.relu,
    n_hidden_layers=1,
    n_hidden_units=9,
    optimizer=tf.train.AdamOptimizer(0.01)
)

assert(X.get_shape().as_list() == [None, x_train.shape[1]])
assert(Y.get_shape().as_list() == [None, 1])
assert(Y_pred.get_shape().as_list() == [None, 1])

with tf.Session() as sess:
    Y_trained = train(
        sess=sess,
        xs=x_train,
        ys=y_train,
        Y_pred=Y_pred,
        loss=cost,
        optimizer=optim,
        epochs=10000,
        batch_size=50
    )
    
    print(accuracy(
        y_pred=predict(Y_pred=Y_trained, xs=x_train, sess=sess),
        y_target=y_train.values.reshape(-1, 1)
    ))<jupyter_output>Epoch: 10/10000
	 Train Cost: 0.3095165193080902
	 Train Acc: 0.8525280952453613
Epoch: 20/10000
	 Train Cost: 0.5360211730003357
	 Train Acc: 0.8553370833396912
Epoch: 30/10000
	 Train Cost: 0.2810772955417633
	 Train Acc: 0.8567415475845337
Epoch: 40/10000
	 Train Cost: 0.2521934509277344
	 Train Acc: 0.8595505356788635
Epoch: 50/10000
	 Train Cost: 0.26337721943855286
	 Train Acc: 0.8567415475845337
Epoch: 60/10000
	 Train Cost: 0.2721528112888336
	 Train Acc: 0.8623595237731934
Epoch: 70/10000
	 Train Cost: 0.3359316289424896
	 Train Acc: 0.8553370833396912
Epoch: 80/10000
	 Train Cost: 0.33043694496154785
	 Train Acc: 0.8553370833396912
Epoch: 90/10000
	 Train Cost: 0.272748738527298
	 Train Acc: 0.8567415475845337
Epoch: 100/10000
	 Train Cost: 0.2117263823747635
	 Train Acc: 0.8567415475845337
Epoch: 110/10000
	 Train Cost: 0.4485659897327423
	 Train Acc: 0.8553370833396912
Epoch: 120/10000
	 Train Cost: 0.31580430269241333
	 Train Acc: 0.8595505356788635
Epoch: 130/10000
	 Trai[...]
{'fraction_non_alphanumeric': 0.15154984149348363, 'fraction_numerical': 0.056974286720676295, 'mean_word_length': 3.6468903436988542, 'pattern_counts': {'":': 0, '<': 59, '<?xml version=': 0, '>': 59, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '518987', 'license': 'MIT', 'n_tokens_mistral': 4788, 'n_tokens_neox': 4227, 'n_words': 742, 'path': 'titanic.ipynb', 'repo_name': 'redouane-dev/titanic-death-prediction', 'size': '11356'}