<jupyter_start><jupyter_text>Titanic -- Tutorial  Let's play the Titanic tutorial and see who survives! In this notebook, we collect some of the code used in the Kaggle tutorials https://www.kaggle.com/c/titanic/details/getting-started-with-python-ii and https://www.kaggle.com/c/titanic/details/getting-started-with-random-forests.<jupyter_code>#seaborn generates pretty plots
import seaborn as sns

#label-encoders turn categories into numbers
from sklearn.preprocessing import LabelEncoder

#Random forest is a basic general-purpose classifier
from sklearn.ensemble import RandomForestClassifier 

#magic command to show plots in notebook
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Load train and test data  and have a look<jupyter_code>#pandas makes it easy to organize our data in data frames
import pandas as pd

train = pd.read_csv('./train.csv', index_col = 0)
train.head()
test = pd.read_csv('./test.csv', index_col = 0)
test.head()<jupyter_output><empty_output><jupyter_text>We merge the train and test data in order to perform feature transformations on both datasets simultaneously.<jupyter_code>train_test = pd.concat([train.drop('Survived', axis = 1), test])<jupyter_output><empty_output><jupyter_text>Play with data aka exploratory data analysis What is the proportion of passengers who survived the disaster?<jupyter_code>train['Survived'].sum()/train.shape[0]<jupyter_output><empty_output><jupyter_text>Let's look at more general statistics for numeric fields<jupyter_code>train.describe()<jupyter_output><empty_output><jupyter_text>How does the survival rate differ from women to men?<jupyter_code>#filter males and females 
train_female = train[train['Sex'] == 'female']
train_male = train[train['Sex'] == 'male']

surv_rate_f = train_female['Survived'].sum()/train_female.shape[0]
surv_rate_m = train_male['Survived'].sum()/train_male.shape[0]

print('Survival rate for females {:0.2f}'.format(surv_rate_f))
print('Survival rate for males {:0.2f}'.format(surv_rate_m))<jupyter_output>Survival rate for females 0.74
Survival rate for males 0.19<jupyter_text>What is the age distribution?<jupyter_code>train_test['Age'].plot('hist', bins=16, title = 'Age distribution')<jupyter_output><empty_output><jupyter_text>Cleaning the data We observe that the age column has missing values.<jupyter_code>train_test['Age'].shape[0] - train_test['Age'].count()<jupyter_output><empty_output><jupyter_text>We fill the missing values by the median age. The same is done for the fare.<jupyter_code>train_test['Age'] = train_test['Age'].fillna(train_test['Age'].median())
train_test['Fare'] = train_test['Fare'].fillna(train_test['Fare'].median())<jupyter_output><empty_output><jupyter_text>Furthermore, the gender should be encoded to a binary numeric value.<jupyter_code>train_test['Sex'] = LabelEncoder().fit_transform(train_test['Sex'])<jupyter_output><empty_output><jupyter_text>Feature Engineering Feature engineering is the process of generating new features of high predictive potential from the given data. For instance, the family size could be of interest.<jupyter_code>train_test['Fam_size'] = train_test['SibSp'] + train_test['Parch']<jupyter_output><empty_output><jupyter_text>Furthermore, we add a feature capturing the two-way interaction between age and passenger class.<jupyter_code>train_test['Age * Class'] = train_test['Age'] + train_test['Pclass']<jupyter_output><empty_output><jupyter_text>Training of machine-learning classifier After having completed the feature engineering, a machine-learning classifier can be fitted to the data. We choose a *random-forest model*, an off-the-shelf classifier providing solid performance for many data sets.<jupyter_code>rf_clf = RandomForestClassifier(n_estimators = 100)
df_num = train_test[['Pclass', 'Sex', 'Age', 'SibSp', 'Parch', 'Fare', 'Fam_size', 'Age * Class']]
rf_clf = rf_clf.fit(df_num.iloc[:train.shape[0], :], train['Survived'])<jupyter_output><empty_output><jupyter_text>After fitting to the numeric columns, we can generate predictions on the test set.<jupyter_code>predicted = rf_clf.predict(df_num.iloc[train.shape[0]:, :])
predicted_series = pd.Series(predicted, index = df_num.iloc[train.shape[0]:, :].index, name = 'Survived')<jupyter_output><empty_output><jupyter_text>Finally, persist the model to disk.<jupyter_code>predicted_series.to_csv('./submission.csv', index = True, header = True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11909604519774011, 'fraction_numerical': 0.005649717514124294, 'mean_word_length': 7.869739478957916, 'pattern_counts': {'":': 0, '<': 60, '<?xml version=': 0, '>': 60, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '582388', 'license': 'Apache-2.0', 'n_tokens_mistral': 1510, 'n_tokens_neox': 1424, 'n_words': 353, 'path': 'Titanic.ipynb', 'repo_name': 'Christian-Hirsch/kaggle_munich_titanic', 'size': '4425'}