<jupyter_start><jupyter_text>**Author:** [<NAME>](http://kdd.isti.cnr.it/people/riccardo-guidotti)  **Python version:**  3.x  Titanic Case StudyThe sinking of the Titanic is one of the most infamous shipwrecks in history. On April 15, 1912, during her maiden voyage, the Titanic sank after colliding with an iceberg, killing 1502 out of 2224 passengers and crew. This sensational tragedy shocked the international community and led to better safety regulations for ships.One of the reasons that the shipwreck led to such loss of life was that there were not enough lifeboats for the passengers and crew. Although there was some element of luck involved in surviving the sinking, some groups of people were more likely to survive than others, such as women, children, and the upper-class.In this notebook, we will try to figure out what sorts of people were likely to survive by using ***clustering*** in python.The **Titanic** dataset became famous after that ***Kaggle*** launched the competition to discover label the passengers as survived or not by exploiting some available features ([link](https://www.kaggle.com/c/titanic)).  Library import  Dataset load and features semantics<jupyter_code>%matplotlib inline
import numpy as np
import pandas as pd
import scipy.stats as stats
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>KDD Process1. Dataset load and features semantics1. Data Cleaning (handle missing values, remove useless variables)1. Feature Engineering1. Clustering Preprocessing (feature reshaping, normalization)1. Parameter Tuning1. Perform Clustering1. Analyze the clustering results1. Are the clusters useful to understand survival?<jupyter_code>df = pd.read_csv("../dataset/titanic.csv", skipinitialspace=True, sep=',')
df.head()
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 12 columns):
PassengerId    891 non-null int64
Survived       891 non-null int64
Pclass         891 non-null int64
Name           891 non-null object
Sex            891 non-null object
Age            714 non-null float64
SibSp          891 non-null int64
Parch          891 non-null int64
Ticket         891 non-null object
Fare           891 non-null float64
Cabin          204 non-null object
Embarked       889 non-null object
dtypes: float64(2), int64(5), object(5)
memory usage: 83.6+ KB<jupyter_text>Each record is described by 12 variables:* The ``Survived`` variable is our outcome or dependent variable. It is a binary nominal datatype of 1 for survived and 0 for did not survive. All other variables are potential predictor or independent variables. It's important to note, more predictor variables do not make a better model, but the right variables.* The ``PassengerID`` and ``Ticket`` variables are assumed to be random unique identifiers, that have no impact on the outcome variable. Thus, they will be excluded from analysis.* The ``Pclass`` variable is an ordinal datatype for the ticket class, a proxy for socio-economic status (SES), representing 1 = upper class, 2 = middle class, and 3 = lower class.* The ``Name`` variable is a nominal datatype. It could be used in feature engineering to derive the gender from title, family size from surname, and SES from titles like doctor or master. Since these variables already exist, we'll make use of it to see if title, like master, makes a difference.* The ``Sex`` and ``Embarked`` variables are a nominal datatype. They will be converted to dummy variables for mathematical calculations.* The ``Age`` and ``Fare`` variable are continuous quantitative datatypes.* The ``SibSp`` represents number of related siblings/spouse aboard and ``Parch`` represents number of related parents/children aboard. Both are discrete quantitative datatypes. This can be used for feature engineering to create a family size and is alone variable.* The ``Cabin`` variable is a nominal datatype that can be used in feature engineering for approximate position on ship when the incident occurred and SES from deck levels. However, since there are many null values, it does not add value and thus is excluded from analysis.<jupyter_code>df.describe(include='all')<jupyter_output><empty_output><jupyter_text>Data cleaning Handle missing values<jupyter_code>df.isnull().sum()
df['Embarked'] = df['Embarked'].fillna(df['Embarked'].mode()[0])

#df['Age'] = df['Age'].fillna(df['Age'].median(), inplace=True)
df['Age'] = df['Age'].groupby([df['Sex'], df['Pclass']]).apply(lambda x: x.fillna(x.median()))<jupyter_output><empty_output><jupyter_text>Remove useless variables<jupyter_code>survived = df['Survived']

column2drop = ['PassengerId', 'Name', 'Cabin', 'Embarked', 'Survived']
df.drop(column2drop, axis=1, inplace=True)
df.head()<jupyter_output><empty_output><jupyter_text>Feature Engineering<jupyter_code>df.boxplot(['Fare'], by='Pclass', showfliers=True)
plt.show()<jupyter_output>/Users/riccardo/anaconda3/lib/python3.6/site-packages/numpy/core/fromnumeric.py:57: FutureWarning: reshape is deprecated and will raise in a subsequent release. Please use .values.reshape(...) instead
  return getattr(obj, method)(*args, **kwds)<jupyter_text>From a carefull data analysis emerges that the Fare is cumulative with respect to various Tickets with the same identifier. Consequently we proceed in correcting the Fare of each passenger by dividing it with the number of Tickets with the same identifier.<jupyter_code>dfnt = df[['Pclass', 'Ticket']].groupby(['Ticket']).count().reset_index()
dfnt['NumTickets'] = dfnt['Pclass']
dfnt.drop(['Pclass'], axis=1, inplace=True)
dfnt.head()
df = df.join(dfnt.set_index('Ticket'), on='Ticket')
df.head()
df['Fare'] = df['Fare']/df['NumTickets']<jupyter_output><empty_output><jupyter_text>It is also important to consider the number of people a passenger is travelling with.<jupyter_code>df['FamilySize'] = df['SibSp'] + df['Parch'] + 1<jupyter_output><empty_output><jupyter_text>The features NumTickets and Ticket helped in correcting Fare and we do not need theme any more.  The features SibSp and Parch are redundant with respect to FamilySize.<jupyter_code>column2drop = ['SibSp', 'Parch', 'NumTickets', 'Ticket']
df.drop(column2drop, axis=1, inplace=True)
df.head()<jupyter_output><empty_output><jupyter_text>Clustering Preprocessing Feature Reshaping<jupyter_code>from sklearn.preprocessing import LabelEncoder
label_encoders = dict()
column2encode = ['Sex']

for col in column2encode:
    le = LabelEncoder()
    df[col] = le.fit_transform(df[col])
    label_encoders[col] = le
df.head()<jupyter_output><empty_output><jupyter_text>Normalization<jupyter_code>from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()
X = scaler.fit_transform(df.values)
X[:5]<jupyter_output><empty_output><jupyter_text>Parameter Tuning<jupyter_code>from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
sse_list = list()
max_k = 50
for k in range(2, max_k + 1):
    kmeans = KMeans(n_clusters=k, n_init=10, max_iter=100)
    kmeans.fit(X)
    
    sse = kmeans.inertia_
    sse_list.append(sse)
plt.plot(range(2, len(sse_list) + 2), sse_list)
plt.ylabel('SSE', fontsize=22)
plt.xlabel('K', fontsize=22)
plt.xticks(np.arange(0, 50, 5))
plt.tick_params(axis='both', which='major', labelsize=22)
plt.show()<jupyter_output><empty_output><jupyter_text>Perform Clustering<jupyter_code>kmeans = KMeans(n_clusters=5, n_init=10, max_iter=100)
kmeans.fit(X)
print('Silhouette %s' % silhouette_score(X, kmeans.labels_))<jupyter_output>Silhouette 0.585626856046<jupyter_text>Analyze the clustering results Observing the size of each cluster<jupyter_code>hist, bins = np.histogram(kmeans.labels_, bins=range(0, len(set(kmeans.labels_)) + 1))
sizes = dict(zip(bins, hist))

for id_cluster, size in sizes.items():
    print('Cluster %d: %d (%.2f)' % (id_cluster, size, size/len(X)))<jupyter_output>Cluster 0: 347 (0.39)
Cluster 1: 170 (0.19)
Cluster 2: 122 (0.14)
Cluster 3: 144 (0.16)
Cluster 4: 108 (0.12)<jupyter_text>Visualization of the clustering in two dimensions.<jupyter_code>plt.scatter(df['Age'], df['Fare'], c=kmeans.labels_, s=20)
plt.tick_params(axis='both', which='major', labelsize=22)
plt.show()<jupyter_output><empty_output><jupyter_text>Centers<jupyter_code>centers = kmeans.cluster_centers_

plt.figure(figsize=(8, 4))
for i in range(0, len(centers)):
    plt.plot(centers[i], marker='o', label='Cluster %s' % i)
plt.tick_params(axis='both', which='major', labelsize=22)
plt.xticks(range(0, len(df.columns)), df.columns, fontsize=18)
plt.legend(fontsize=16, ncol=2, bbox_to_anchor=(1.1, 1.05))
plt.show()
pd.crosstab(df['Pclass'], kmeans.labels_)
pd.crosstab(df['Sex'], kmeans.labels_)<jupyter_output><empty_output><jupyter_text>Are the clusters useful to understand survival?<jupyter_code>df['Survived'] = survived
df['Label'] = kmeans.labels_

for col in column2encode:
    le = label_encoders[col]
    df[col] = le.inverse_transform(df[col])
pd.crosstab(df['Survived'], df['Label'])
df[df['Label']==4].describe(include='all')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11493492168541805, 'fraction_numerical': 0.02382528127068167, 'mean_word_length': 5.963901689708141, 'pattern_counts': {'":': 0, '<': 67, '<?xml version=': 0, '>': 67, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '58294', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 3081, 'n_tokens_neox': 2844, 'n_words': 914, 'path': 'Intro_to_data_analysis_with_python/5. Case Study - Titanic Clustering.ipynb', 'repo_name': 'GiulioRossetti/SoBigData_courses', 'size': '9077'}