<jupyter_start><jupyter_text>Titanic Prediction using Python A huge thank you to <NAME> and his Udemy course for teaching me https://www.udemy.com/python-for-data-science-and-machine-learning-bootcamp/learn/v4  Imports and reading in files<jupyter_code>import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

from subprocess import check_output
#print(check_output(["ls", "../input"]).decode("utf8"))

import warnings
warnings.filterwarnings('ignore')

df = pd.read_csv("../input/train.csv")
#Check that the file was read in properly and explore the columns
df.head()<jupyter_output><empty_output><jupyter_text>Data exploration<jupyter_code>plt.figure(figsize=(12,8))
sns.heatmap(df.isnull(),cbar=False, yticklabels=False, cmap='viridis')<jupyter_output><empty_output><jupyter_text>From the heat map, we can see that a lot of the 'Cabin' row information is missing. However, while the age column is also missing some data, we can use imputation to fill in some of the data later. Additionally, the 'Embarked' column has so few rows missing, that we can just delete those.<jupyter_code>sns.set_style('darkgrid')
sns.countplot(x='Survived', data=df, hue='Pclass')<jupyter_output><empty_output><jupyter_text>We can see here that those who did not survive were predominantly from the 3rd Passenger Class (Pclass).<jupyter_code>sns.countplot(x='SibSp', data=df, hue='Survived')
df['Fare'].hist(bins=40)<jupyter_output><empty_output><jupyter_text>Here, we impute the age of those we do not have information on. We use a boxplot to estimate the median age of each class, and impute that into the age for the rows with missing age.<jupyter_code>plt.figure(figsize=(12,6))
sns.boxplot(x='Pclass', y='Age', data=df)
def inpute_age(cols):
    Age = cols[0]
    Pclass = cols[1]
    
    if pd.isnull(Age):
        if Pclass == 1:
            return 37
        elif Pclass == 2:
            return 29
        else: return 24
    else: return Age
df['Age']=df[['Age','Pclass']].apply(inpute_age, axis=1)
plt.figure(figsize=(12,8))
sns.heatmap(df.isnull(),cbar=False, yticklabels=False, cmap='viridis')<jupyter_output><empty_output><jupyter_text>You can see now the data is cleaner, but we still need to clean the 'Cabin' and 'Embarked' columns. For now, we will simply drop the 'Cabin' column and drop the rows where 'Embarked' is missing.<jupyter_code>df.drop('Cabin', axis=1, inplace=True)
plt.figure(figsize=(12,6))
sns.heatmap(df.isnull(),cbar=False, yticklabels=False, cmap='viridis')
df.dropna(inplace=True)
plt.figure(figsize=(12,6))
sns.heatmap(df.isnull(),cbar=False, yticklabels=False, cmap='viridis')<jupyter_output><empty_output><jupyter_text>The data is now clean of null values, but we still need to take care of objects that a machine learning algorithm can't handle, namely strings.<jupyter_code>df.info()<jupyter_output><empty_output><jupyter_text>We can see that 'Name', 'Sex', 'Ticket', and 'Embarked' are all objects. In this case, they indeed are all strings. We will use Pandas built in getDummies() funciton to convert those to numbers.<jupyter_code>#We make a new 'Male columns because getDummies will drop one the the dummy variables
#to ensure linear independence.
df['Male'] = pd.get_dummies(df['Sex'], drop_first=True)
#The embarked column indicates where the passenger boarded the Titanic.
#It has three values ['S','C','Q']
embarked = pd.get_dummies(df['Embarked'], drop_first=True)
df = pd.concat([df, embarked], axis=1)
#These columns do not provide us any information for the following reasons:
#PassengerID: we consider 'PassengerID' a randomly assigned ID thus not correlated with surviability
#Name: we are not performing any feature extraction from the name, so we must drop tihs non-numerical column
#Sex: the 'Male' column already captures all information about the sex of the passenger
#Ticket: we are not performing any feature extraction, so we must drop this non-numerical column
#Embarked: we have extracted the dummy values, so those two numerical dummy values encapsulate all the embarked info

df.drop(['PassengerId', 'Name', 'Sex', 'Ticket', 'Embarked'], axis=1, inplace=True)
#Take a look at our new dataframe
df.head()
df.info()<jupyter_output><empty_output><jupyter_text>Build and train the model<jupyter_code>#Seperate the feature columns from the target column
X = df.drop('Survived', axis=1)
y = df['Survived']
#Split the data into two. I don't think this is necessary since there are two files.
#I will keep this here for now
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.3)
from sklearn.linear_model import LogisticRegression
logmodel = LogisticRegression()
logmodel.fit(X, y)
#Read in the test data
test_df = pd.read_csv('../input/test.csv')
#Clean the test data the same way we did the training data
test_df['Age']=test_df[['Age','Pclass']].apply(inpute_age, axis=1)
test_df.drop('Cabin', axis=1, inplace=True)
test_df.dropna(inplace=True)
test_df['Male'] = pd.get_dummies(test_df['Sex'], drop_first=True)
embarked = pd.get_dummies(test_df['Embarked'], drop_first=True)
test_df = pd.concat([test_df, embarked], axis=1)
pass_ids = test_df['PassengerId']
test_df.drop(['PassengerId', 'Name', 'Sex', 'Ticket', 'Embarked'], axis=1, inplace=True)
test_df.tail()
predictions = logmodel.predict(test_df)
submission = pd.DataFrame({
        "PassengerId": pass_ids,
        "Survived": predictions
    })
submission.to_csv('titanic.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11783783783783784, 'fraction_numerical': 0.007387387387387387, 'mean_word_length': 6.0444162436548226, 'pattern_counts': {'":': 2, '<': 38, '<?xml version=': 0, '>': 38, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '35499', 'license': 'MIT', 'n_tokens_mistral': 1912, 'n_tokens_neox': 1836, 'n_words': 577, 'path': '#4/notebooks/.ipynb_checkpoints/Titanic_logistic-checkpoint.ipynb', 'repo_name': 'Liberus/AIClub', 'size': '5557'}