<jupyter_start><jupyter_text>Problem 02: Preparing the Titanic Dataset**Author:** <NAME>--- In this problem, we use de CRISP-DS methodology to clean, prepare and analyze the well-known Titanic dataset. CRISP-DS is an adaptation for Data Science of the cyclic methodology CRISP-DM (_Cross Industry Standard Process for Data Mining_). [[1]](https://blog.magrathealabs.com/crisp-ds-cyclic-methodology-for-data-science-projects-10c7d00fbc85)These are the steps involved in the CRISP-DS methodology:![](https://raw.githubusercontent.com/diascarolina/i2a2/main/img/crisp-ds.png)Since this is a fairly straightforward problem, we only need to go through the cycle once.Our **Business Problem** is to predict whether a passenger survived the sinking of Titanic. This is a famous dataset generally used to learn about Data Science and Machine Learning, and it can also be found at Kaggle:- [Kaggle Problem: "Titanic - Machine Learning from Disaster"](https://www.kaggle.com/c/titanic)  1 Business UnderstandingFrom the Kaggle Problem page:> The sinking of the Titanic is one of the most infamous shipwrecks in history.>> On April 15, 1912, during her maiden voyage, the widely considered “unsinkable” RMS Titanic sank after colliding with an iceberg. Unfortunately, there weren’t enough lifeboats for everyone onboard, resulting in the death of 1502 out of 2224 passengers and crew.>> While there was some element of luck involved in surviving, it seems some groups of people were more likely to survive than others.  2 Data ExtractionWe don't need to extract this data, since it is already given to us. So, in this step, let's import the necessary libraries and load up the data into the notebook.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

from sklearn.ensemble import RandomForestClassifier
df_train_raw = pd.read_excel('https://github.com/diascarolina/i2a2/blob/main/exercise02/train.xlsx?raw=true')
df_test_raw = pd.read_excel('https://github.com/diascarolina/i2a2/blob/main/exercise02/test.xlsx?raw=true')<jupyter_output><empty_output><jupyter_text>3 Data Cleaning & EDALet's first understand the data that we have here by checking what each column means by looking at its data dictionary.- **Kaggle_Pass_Id**: the id of a person (used on Kaggle)- **pclass**: ticket class,	1 = 1st, 2 = 2nd, 3 = 3rd- **survived**: whether someone survived or not (0 no, 1 yes)- **name**: the name of a person- **sex**: the gender of a person- **age**: the age of a person- **sibsp**: number of siblings / spouses aboard the Titanic- **parch**: number of parents / children aboard the Titanic- **ticket**: ticket number- **fare**: passenger fare- **cabin**: cabin number- **embarked**: port of embarkation, C = Cherbourg, Q = Queenstown, S = SouthamptonThe following information are not provided in the data dictionary:- **boat**- **body**- **home.dest**- **orig_seq**  Let's check the first five lines of the datasets (train and test), to get a general look for the data.<jupyter_code>df_train = df_train_raw.copy()
df_test = df_test_raw.copy()
df_train.head()
df_test.head()
print(f'We have {df_train.shape[0]} rows and {df_train.shape[1]} columns in the train dataset.')
print(f'We have {df_test.shape[0]} rows and {df_test.shape[1]} columns in the test dataset.')<jupyter_output>We have 891 rows and 16 columns in the train dataset.
We have 418 rows and 15 columns in the test dataset.<jupyter_text>We can see that in the `test` dataset we have one less column than in the `train` dataset. The column missing is the `survived` columns, because it is what we have to predict. But, in my opinion, this column should be provided for us to compare our results with the real prediction. What is the data type of the columns?<jupyter_code>df_train.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 16 columns):
 #   Column          Non-Null Count  Dtype  
---  ------          --------------  -----  
 0   Kaggle_Pass_Id  891 non-null    int64  
 1   pclass          891 non-null    int64  
 2   survived        891 non-null    int64  
 3   name            891 non-null    object 
 4   sex             891 non-null    object 
 5   age             714 non-null    float64
 6   sibsp           891 non-null    int64  
 7   parch           891 non-null    int64  
 8   ticket          891 non-null    object 
 9   fare            891 non-null    float64
 10  cabin           204 non-null    object 
 11  embarked        889 non-null    object 
 12  boat            330 non-null    object 
 13  body            83 non-null     float64
 14  home.dest       499 non-null    object 
 15  orig_seq        891 non-null    int64  
dtypes: float64(3), int64(6), object(7)
memory usage: 111.5+ KB<jupyter_text>What about missing values in the data?<jupyter_code>df_train.isna().sum()<jupyter_output><empty_output><jupyter_text>From this we can see that a lot of columns have missing data. What should we do about them?Here, the only variable we can treat the null values is the `age` columns. The other columns have the majority of data as null values, so we'll drop these columns.<jupyter_code>df_train = df_train.drop(['cabin', 'boat', 'body', 'home.dest'], axis = 1)
df_train.isna().sum()<jupyter_output><empty_output><jupyter_text>We still have 2 missing values in the column `embarked`. Since this is a small number, we can change the null values to the most common one.<jupyter_code>df_train.embarked.value_counts()
df_train.embarked[df_train.embarked.isna()] = 'S'
df_train.isna().sum()<jupyter_output><empty_output><jupyter_text>Now, for the `age` column, we'll substitute the missing values with the mean.<jupyter_code>df_train.age = df_train.age.fillna(df_train.age.mean())
df_train.isna().sum()<jupyter_output><empty_output><jupyter_text>Let's check the statistical part of the data, but first, let's change the values in the `sex` columns such as:- **female**: 0- **male**: 1<jupyter_code>df_train.sex = df_train.sex.map({'female': '0', 'male': '1'}).astype(float)
df_train.describe()
plt.figure(figsize = (12, 8))
sns.heatmap(df_train.corr(), cmap = "BuPu", annot = True)
plt.xticks(fontsize = 14)
plt.yticks(fontsize = 14)
plt.title('How is the data correlated to each other?', fontdict = {'fontsize': 25}, loc = 'left')
sns.despine()
plt.show()<jupyter_output><empty_output><jupyter_text>From the above heatmap we see that the variables that most influence the `survived` target are:- sex- pclass- fare- orig_seqDoes this make sense from our point of view?Yes! It is well-known that when the Titanic was sinking the first people saved were "women and children first". So it makes sense that most women survived instead of men. It is also surprising that age dosn't have that high of a correlation with the survived target.Also, `pclass`, `fare` and `orig_seq` have high correlation with the survived class because they are variables related to the socio-economic class of a person, and people in higher classes were saved first.  3.1 Variables CheckNow let's check visually for the distribution of variables.<jupyter_code>df_train.pclass.value_counts().plot(kind = 'bar', figsize = (10, 6), color = 'seagreen')
plt.title('Ticket Class (pclass)', fontsize = 22)
plt.xticks(rotation = 0, fontsize = 15)
plt.yticks(fontsize = 15)
plt.xlabel('Class', fontsize = 20)
plt.ylabel('Count', fontsize = 18)
sns.despine()
plt.show()<jupyter_output><empty_output><jupyter_text>More people were in the 3rd class.<jupyter_code>df_train.survived.value_counts().plot(kind = 'bar', figsize = (10, 6), color = 'seagreen')
plt.title('How many survived?', fontsize = 22)
plt.xticks(rotation = 0, fontsize = 15)
plt.yticks(fontsize = 15)
plt.xlabel('Survived', fontsize = 20)
plt.ylabel('Count', fontsize = 18)
sns.despine()
plt.show()<jupyter_output><empty_output><jupyter_text>More people died than survived.<jupyter_code>df_train.sex.value_counts().plot(kind = 'bar', figsize = (10, 6), color = 'seagreen')
plt.title('Gender Distribution', fontsize = 22)
plt.xticks(rotation = 0, fontsize = 15)
plt.yticks(fontsize = 15)
plt.xlabel('Gender', fontsize = 20)
plt.ylabel('Count', fontsize = 18)
sns.despine()
plt.show()<jupyter_output><empty_output><jupyter_text>More men than women in the Titanic.<jupyter_code>plt.figure(figsize = (12, 5))
plt.title("Age Distribution", fontdict = {'fontsize': 24})
sns.set_style("white")
ax = sns.histplot(df_train.age, kde = True, color = 'seagreen')
plt.xticks(fontsize = 14)
plt.yticks(fontsize = 14)
plt.xlabel('Age', fontsize = 15)
plt.ylabel('Count', fontsize = 15)
sns.despine()
plt.show()<jupyter_output><empty_output><jupyter_text>The majority of people were about 30 years of age.  4 Data PreparationFrom the above analysis we can see that some variables are more relavant than other in the prediction of the survived target. So let's take the other variables out of the dataframe. Let's fill the missing values as well, as we did above.<jupyter_code>def data_preprocessing(df):
    selected_columns = ['pclass', 'sex', 'fare']
    df.sex = df.sex.map({'female': '0', 'male': '1'}).astype(float)
    return df[selected_columns]
X_train = data_preprocessing(df_train_raw)
y_train = df_train.survived

X_test = data_preprocessing(df_test_raw)
X_train.isna().sum()
X_test.isna().sum()
X_test.fare = X_test.fare.fillna(X_test.fare.mean())<jupyter_output>/usr/local/lib/python3.7/dist-packages/pandas/core/generic.py:5170: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  self[name] = value<jupyter_text>5 Modelling For this part, we'll be using one of the most famous and used model: the Random Forest Classifier. Most of the time, its results are good enough for what we want to do. The won't be any hyperparameter tuning.We could also use a dummy model to check what the accuracy would be in the worst case scenario, by predicting that everyone didn't survive.<jupyter_code># initializing, fitting and predicting with the model
rfc = RandomForestClassifier(random_state=42)
rfc.fit(X_train, y_train)
y_preds = rfc.predict(X_test)<jupyter_output><empty_output><jupyter_text>To evaluate our model we need to have the `survived` column in the test set. Since the provided data doesn't have it, we'll be using the Kaggle competition page to evaluate it.<jupyter_code># preparing the submission in the Kaggle format
submission = pd.DataFrame()
submission['PassengerId'] = df_test_raw['Kaggle_Pass_Id']
submission['Survived'] = y_preds

# uncomment the following line to download the .csv
# submission.to_csv('submission.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10603543743078626, 'fraction_numerical': 0.023348098929494278, 'mean_word_length': 4.617936754795231, 'pattern_counts': {'":': 0, '<': 60, '<?xml version=': 0, '>': 65, 'https://': 6, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '152209', 'license': 'MIT', 'n_tokens_mistral': 3691, 'n_tokens_neox': 3383, 'n_words': 1225, 'path': 'exercise02/problem02.ipynb', 'repo_name': 'diascarolina/i2a2', 'size': '10850'}