<jupyter_start><jupyter_text>Data Descriptions | Variable      | Definition    | Key     || :------------- | :----------: | -----------: ||  **survival** |  Survival  |  0 = No, 1 = Yes   || **pclass**  | Ticket class | 1 = 1st, 2 = 2nd, 3 = 3rd||  **sex** | Sex   |     ||  **Age**  | Age in years |  ||  **sibsp** |   of siblings / spouses aboard the Titanic  |     ||  **parch**  |  of parents / children aboard the Titanic |  ||  **ticket** |  Ticket number  |     ||  **fare**  | Passenger fare |  ||  **cabin**  | Cabin Number |   ||  **embarked**  | Port of Embarkation | C = Cherbourg, Q = Queenstown,S = Southampton |<jupyter_code>train.head()
len(train)
train.describe()
# Datatype and info about train dataframe
train.info()
test.head()
len(test)
test.describe()
# Datatype and info about test dataframe
test.info()
gender_submission.head()
len(gender_submission)<jupyter_output><empty_output><jupyter_text>Missing Values<jupyter_code># Counting the no. of missing values 
train.isnull().sum()
# Visualisng the missing values
plt.figure(figsize = (7,3), dpi = 130)
ax = sns.heatmap(train.isnull(), cmap = "Oranges")


tick = [train['PassengerId'].min(), train['PassengerId'].max()]
plt.yticks(tick,tick, fontsize = 7)
plt.xticks(fontsize = 7)

for i in range(train.shape[1]+1):
    ax.axvline(i, color='white', lw = 2)

plt.show()<jupyter_output><empty_output><jupyter_text>We see that we are missing almost 20% data in Age column and almost 80% data in Cabin column. So its best to drop the cabin column later in our analysis or change it to "Cabin known or unknown".  Univariate analysis  Creating a function to create countplot<jupyter_code>def plot_count(target_column, data, hue=None, palette=None, color=None, dpi=None, figsize=(6, 6)):
    plt.figure(figsize=figsize, dpi=dpi)
    sns.set_style('darkgrid')
    sns.countplot(x=target_column, data=data, hue=hue, palette=palette, color=color)
    plt.show()<jupyter_output><empty_output><jupyter_text>Feature : Survived Description: Whether the passenger survived or not. Key: 0 = did not survive, 1 = survived This is the variable we want our machine learning model to predict based off all the others.  Of all the people in training set how many survived and how many died ?<jupyter_code>train.head()
# How many people survived
train["Survived"].value_counts()
# Visualising how many people survived
plot_count("Survived", train, figsize=(5,6), palette="rainbow")<jupyter_output><empty_output><jupyter_text>Feature: Pclass Description: The ticket class of the passenger. Key: 1 = 1st Class, 2 = 2nd Class, 3 = 3rd Class  How many passengers in each Pclass ?<jupyter_code># Count the no. of passengers in different classes 
plot_count("Pclass", train, color="#ff6666")<jupyter_output><empty_output><jupyter_text>We see that there were a lot of passengers who belonged to 3rd class  Feature: Name Description: The name of the passenger.<jupyter_code>train.head()
train["Name"].value_counts()<jupyter_output><empty_output><jupyter_text>We see that every name occurs uniquely. What we can do is extract a different titles from the names and create a new column based on it First we can see the different titles in the list<jupyter_code># Collecting different titles in the list
names = []
for name in train["Name"]:
    names.append(name.split(",")[1].split(".")[0].strip())
names = list(set(names))
print(names)<jupyter_output>['Mrs', 'Mlle', 'the Countess', 'Mr', 'Miss', 'Mme', 'Col', 'Jonkheer', 'Dr', 'Sir', 'Master', 'Capt', 'Don', 'Major', 'Ms', 'Rev', 'Lady']<jupyter_text>Now making column for the titles<jupyter_code>train["Title"] = train.Name.apply(lambda name: name.split(",")[1].split(".")[0].strip())
test["Title"] = test.Name.apply(lambda name: name.split(",")[1].split(".")[0].strip())

test.head()<jupyter_output><empty_output><jupyter_text>So we can drop the name column in our prediction and replace it with Title column instead.<jupyter_code>train["Title"].value_counts()<jupyter_output><empty_output><jupyter_text>Additionaly we can also replace title with royalty based on the titles of people  Feature: Sex Description: The sex of the passenger (male or female)<jupyter_code># No. of males and females
train["Sex"].value_counts()
# Visualisng No of males and females
plot_count("Sex", train, figsize=(4,5), dpi=100)<jupyter_output><empty_output><jupyter_text>Feature: Age Description: Tells age of passenger<jupyter_code>train.head()
train.Age.isnull().sum()<jupyter_output><empty_output><jupyter_text>We see that we have some missing values in age columns. So we cant do much here.Checking correlation of survived and age column:-<jupyter_code>print('Oldest Passenger was of:',train['Age'].max(),'Years')
print('Youngest Passenger was of:',train['Age'].min(),'Years')
print('Average Age on the ship:',train['Age'].mean(),'Years')
train["Survived"].corr(train["Age"])<jupyter_output><empty_output><jupyter_text>We see that we have a negative correlation i.e. You have a higher chance of death if you were older. We can create a bin for the known ages and predict survival rates based on it or we can replace the missing ages by median of agesFor now we will bin the ages and create a new column based on it<jupyter_code># Binning the ages
bins = [0, 20, 40, 60, 100]
labels =[1, 2, 3, 4]

train["Age_Bin"] = pd.cut(train["Age"], bins=bins, labels=labels)
test["Age_Bin"] = pd.cut(test["Age"], bins=bins, labels=labels)
test.head()
train.Age_Bin.value_counts()
# Visualising Bin counts
plot_count("Age_Bin", train, dpi=100, figsize=(5,5))<jupyter_output><empty_output><jupyter_text>Features: SibSp and Parch Description: SibSp - The number of siblings/spouses the passenger has aboard the Titanic.             Parch - The number of parents/children the passenger has aboard the Titanic.Since both the features are similar we'll do a similar analysis.<jupyter_code># Checking for null values
train.Parch.isnull().sum()
train.SibSp.isnull().sum()
# Count of Parch and Sibsp
train.Parch.value_counts()
plot_count("Parch", train, dpi=100)
train.SibSp.value_counts()
plot_count("SibSp", train, dpi=100)<jupyter_output><empty_output><jupyter_text>Since both of them are similar we can make a new column out of the two columns named **"Alone"** where **0 = "Alone"** and **1 = "Not Alone"**<jupyter_code># Add Alone column
train["Alone"] = 0
train.loc[(train.SibSp + train.Parch)>0, "Alone"] = 1
train.head()
test["Alone"] = 0
test.loc[(test.SibSp + test.Parch)>0, "Alone"] = 1
test.head()
train.Alone.value_counts()
plot_count("Alone", train)<jupyter_output><empty_output><jupyter_text>Feature: Ticket Description: The ticket number of the boarding passenger.<jupyter_code>train.Ticket.value_counts()
train.Ticket.nunique()<jupyter_output><empty_output><jupyter_text>Since there are so many ticket values its best to drop them  Feature: Fare Description: How much the ticket cost.<jupyter_code>train.Fare.nunique()<jupyter_output><empty_output><jupyter_text>Like age we can bin the fares into a new column.<jupyter_code># Binning the fares
fares = []
for fare_bin in pd.cut(train["Fare"], bins=4):
    fares.append(fare_bin)
fares = tuple(set(fares))
labels = (1,2,3,4)
fare_dict = dict(zip(fares, labels))
fare_dict
# Creating a new binned fare column
train["Fare_Bin"] = pd.cut(train["Fare"], bins=4)
train["Fare_Bin"] = train.Fare_Bin.map(fare_dict)

test["Fare_Bin"] = pd.cut(test["Fare"], bins=4)
test["Fare_Bin"] = test.Fare_Bin.map(fare_dict)
train.head()
train.Fare_Bin.value_counts()
# Visualising binned fare column
plot_count("Fare_Bin", train, dpi=100)<jupyter_output><empty_output><jupyter_text>Feature: Cabin Description: The cabin number where the passenger was staying.<jupyter_code>train.Cabin.isnull().sum()
train.Cabin.value_counts()<jupyter_output><empty_output><jupyter_text>We see that we have a lot of NaN. So the best idea is to drop them. What we can also do is fill the missing values based on Pclass and Gender.For now we will be dropping the column  Feature: Embarked Description: The port where the passenger boarded the Titanic.Key: C = Cherbourg, Q = Queenstown, S = Southampton<jupyter_code>train.Embarked.isnull().sum()
train.Embarked.value_counts()
plot_count("Embarked", train, palette="OrRd_r")<jupyter_output><empty_output><jupyter_text>For the two missing values we can drop those two rows.  Correlation between each columnBefore diving into Bivariate analysis lets plot the correlation between each column**POSITIVE CORRELATION**: If an increase in feature A leads to increase in feature B, then they are positively correlated. A value 1 means perfect positive correlation.**NEGATIVE CORRELATION**: If an increase in feature A leads to decrease in feature B, then they are negatively correlated. A value -1 means perfect negative correlation<jupyter_code>plt.figure(figsize=(8,6))
sns.set_context('poster', font_scale=0.5)
data=train.corr()
sns.heatmap(data, annot=True, cmap='coolwarm')
plt.show()<jupyter_output><empty_output><jupyter_text>BiVariate Analysis  Creating a function for plotting bar and catplot<jupyter_code>def plot_bar_cat(x, y, data, hue=None, palette=None, color=None, dpi=None, figsize=(6, 6), kind="strip"):
    sns.set_style('darkgrid')
    f, (ax1, ax2) = plt.subplots(ncols=2, sharey=True, figsize=figsize)
    sns.barplot(x=x, y=y, data=data, hue=hue, palette=palette, color=color, ax=ax1)
    sns.pointplot(x=x, y=y, data=data, hue=hue, palette=palette, color=color, ax=ax2)
    plt.show()<jupyter_output><empty_output><jupyter_text>Relationship between Sex and Survived<jupyter_code>train.head()
# Total no. of men and women who survived or died 
train.groupby(["Sex"])["Survived"].value_counts()
# Building a crosstab
pd.crosstab(index=train['Sex'], columns=train['Survived'], margins=True).style.background_gradient(cmap="GnBu")
# Plotting how many male or female died or survived 
plot_count("Survived", train, hue = "Sex", palette="Accent", figsize=(7,6))
# Percentage of female and male survived vs dead
male_survive = (train.loc[(train["Sex"] == "male") & (train.Survived == 1)].Survived.count())/(train.loc[(train["Sex"] == "male")].Survived.count())
female_survive = (train.loc[(train["Sex"] == "female") & (train.Survived == 1)].Survived.count())/(train.loc[(train["Sex"] == "female")].Survived.count())

print(f"% male survived in the accident are: {male_survive}")
print(f"% female survived in the accident are: {female_survive}")
f, ax = plt.subplots(1, 2, figsize=(10, 9))
male_list = [male_survive, 1-male_survive]
female_list = [female_survive, 1-female_survive]

labels = ["survived", "not survived"]
explode = (0, .2)

ax[0].pie(male_list, explode=explode, labels=labels, autopct='%1.1f%%',
        shadow=True)
ax[1].pie(female_list, explode=explode, labels=labels, autopct='%1.1f%%',
        shadow=True)


ax[0].set_title('Survived (male)')
ax[1].set_title('Survived (female)')

plt.show()<jupyter_output><empty_output><jupyter_text>We see that females are more likely to survive than males.74% of women survive but only 19% of men survive in training set  Relationship between PClass and Survived<jupyter_code>train.head()
# Total passengers who survived according to cabin class
train.groupby(["Survived", "Pclass"]).Survived.count()
# Building a crosstab
pd.crosstab(index=train['Pclass'], columns=train['Survived'], margins=True).style.background_gradient(cmap = "RdPu")
# Plotting the graph of Pclass vs Survived
plot_count("Survived", train, hue="Pclass", palette="Accent", figsize=(7,6))
# Percentage survived passengers according to Pclass
first_class = (train.loc[(train["Pclass"] == 1) & (train.Survived == 1)].Survived.count())/(train.loc[(train["Pclass"] == 1)].Survived.count())
second_class = (train.loc[(train["Pclass"] == 2) & (train.Survived == 1)].Survived.count())/(train.loc[(train["Pclass"] == 2)].Survived.count())
third_class = (train.loc[(train["Pclass"] == 3) & (train.Survived == 1)].Survived.count())/(train.loc[(train["Pclass"] == 3)].Survived.count())

print("% Passengers survived in:-")
print(f"First class are: {first_class}")
print(f"Second class are: {second_class}")
print(f"Third class are: {third_class}")
# Visualing percentage survived passengers
plot_bar_cat(x='Pclass',y='Survived',data=train, kind="point", figsize=(12,6))<jupyter_output><empty_output><jupyter_text>We see that higher class passengers have higher rate of survival   Relationship between Embarked and Survived<jupyter_code>train.head()
pd.crosstab(index=train.Embarked, columns=train.Survived, margins=True).style.background_gradient(cmap = "summer_r")
plot_count("Survived", train, hue="Embarked", figsize=(7,6))
# Percentage survived passengers according to Embarked
Southampton = (train.loc[(train["Embarked"] == "S") & (train.Survived == 1)].Survived.count())/(train.loc[(train["Embarked"] == "S")].Survived.count())
Cherbourg = (train.loc[(train["Embarked"] == "C") & (train.Survived == 1)].Survived.count())/(train.loc[(train["Embarked"] == "C")].Survived.count())
Queenstown = (train.loc[(train["Embarked"] == "Q") & (train.Survived == 1)].Survived.count())/(train.loc[(train["Embarked"] == "Q")].Survived.count())

print("% Passengers survived from:-")
print(f"Southampton are: {Southampton}")
print(f"Cherbourg are: {Cherbourg}")
print(f"Queenstown are: {Queenstown}")
plot_bar_cat(x='Embarked',y='Survived',data=train, figsize=(12,5))<jupyter_output><empty_output><jupyter_text>If you are from Cherbourg you have a higher chance of survival  Relationship between Survived and Age_Bin<jupyter_code>train.head()
pd.crosstab(index=train.Age_Bin, columns=train.Survived, margins=True).style.background_gradient(cmap = "Blues")
plot_count("Survived", train, hue="Age_Bin", figsize=(9,6))
Teens = (train.loc[(train["Age_Bin"] == 1) & (train.Survived == 1)].Survived.count())/(train.loc[(train["Age_Bin"] == 1)].Survived.count())
Youth = (train.loc[(train["Age_Bin"] == 2) & (train.Survived == 1)].Survived.count())/(train.loc[(train["Age_Bin"] == 2)].Survived.count())
Adult = (train.loc[(train["Age_Bin"] == 3) & (train.Survived == 1)].Survived.count())/(train.loc[(train["Age_Bin"] == 3)].Survived.count())
Old = (train.loc[(train["Age_Bin"] == 4) & (train.Survived == 1)].Survived.count())/(train.loc[(train["Age_Bin"] == 4)].Survived.count())

print("% Out of survived passengers:-")
print(f"Teens were: {Teens}")
print(f"Youth were: {Youth}")
print(f"Adult are: {Adult}")
print(f"Old are: {Old}")
plot_bar_cat(x='Age_Bin',y='Survived',data=train, figsize=(12,6))<jupyter_output><empty_output><jupyter_text>Elder people that is 60+ has the least possibility of survival  Relationship between Survived and Alone<jupyter_code>train.head()
pd.crosstab(index=train.Alone, columns=train.Survived, margins=True).style.background_gradient(cmap = "OrRd")
plot_count("Survived", train, hue="Alone", figsize=(6,6))
plot_bar_cat(x='Alone',y='Survived',data=train, figsize=(12,6))<jupyter_output><empty_output><jupyter_text>We see that majority of people who were alone i.e. they didnt have either a sibling/spouse or parents died on titanic  MultiVariate Analysis  Relationship between Pclass, Sex and Survived<jupyter_code>pd.crosstab(index=[train.Sex, train.Survived], columns=train.Pclass, margins=True).style.background_gradient(cmap="hot_r")
sns.catplot(x='Pclass', y='Survived',hue='Sex',data=train, kind="point")
plt.show()<jupyter_output><empty_output><jupyter_text>We see that females who belonged to first class and second class had almost 100% chance of survival as only 3 out of 94 Women from Pclass1 and only 6 of 70 Pclass2 died. While among males first class males had the highest chances of survival. It is evident that irrespective of Pclass, Women were given first priority while rescue. Even Men from Pclass1 have a very low survival rate.  Relationship between Age/Age_Bin and Survived with Pclass and Sex<jupyter_code>f,ax=plt.subplots(1,2,figsize=(18,8))
sns.violinplot(x="Pclass",y="Age", hue="Survived", data=train,split=True,ax=ax[0])
ax[0].set_title('Pclass and Age vs Survived')
ax[0].set_yticks(range(0,110,10))
sns.violinplot(x="Sex",y="Age", hue="Survived", data=train,split=True,ax=ax[1])
ax[1].set_title('Sex and Age vs Survived')
ax[1].set_yticks(range(0,110,10))
plt.show()<jupyter_output><empty_output><jupyter_text>Observations1)The number of children increases with Pclass and the survival rate for passenegers below Age 10(i.e children) looks to be good irrespective of the Pclass.2)Survival chances for Passenegers aged 20-50 from Pclass1 is high and is even better for Women.3)For males, the survival chances decreases with an increase in age.<jupyter_code>pd.crosstab(index=[train.Sex, train.Survived], columns=train.Age_Bin, margins=True).style.background_gradient(cmap="pink_r")
sns.catplot(x='Age_Bin', y='Survived',hue='Sex',data=train, kind="point")
plt.show()<jupyter_output><empty_output><jupyter_text>We see that all the females who were above 60 i.e. 3 out of 22 survived while majority of males, i.e. 17 out of 22 who were above 60 died.Same goes with males who were above 20 and 40. Majority of the people died. Only men between who were below 20 had a higher chance of survival<jupyter_code>pd.crosstab(index=[train.Pclass, train.Survived], columns=train.Age_Bin, margins=True).style.background_gradient(cmap="BuGn")
sns.catplot(x='Age_Bin', y='Survived',hue='Pclass',data=train, kind="point")
plt.show()<jupyter_output><empty_output><jupyter_text>We see that people who were above 60 despite being in the first class (11 out of 22) died.Majority of the people that died and were below 20 belonged to Third class<jupyter_code>train.head()
pd.crosstab(index=[train.Pclass, train.Sex, train.Survived], columns=train.Age_Bin, margins=True).style.background_gradient(cmap="pink_r")<jupyter_output><empty_output><jupyter_text>* We see here that majority males who died were between the age 20-40 and belonged to the third class * We also observe that all the passengers who were above 60 and belonged to third class died and only one out from first class and second class survived* All the females who were in second class and were of age below 20 survived   Relationship between Alone and Survived with Age and Pclass<jupyter_code>pd.crosstab(index=[train.Sex, train.Survived], columns=train.Alone, margins=True).style.background_gradient(cmap="BuPu")
sns.catplot(x='Alone', y='Survived',hue='Sex',data=train, kind="point")
plt.show()<jupyter_output><empty_output><jupyter_text>We see that males who were alone had a very low chance of survival<jupyter_code>pd.crosstab(index=[train.Pclass, train.Survived], columns=train.Alone, margins=True).style.background_gradient(cmap="PuBu")
sns.catplot(x='Alone', y='Survived',hue='Pclass',data=train, kind="point")
plt.show()
pd.crosstab(index=[train.Pclass, train.Survived], columns=[train.Alone, train.Sex], margins=True).style.background_gradient(cmap="PuBu")<jupyter_output><empty_output><jupyter_text>Majority of the males were alone and belonged to third class.  Relationship between Embarked and Survived with Pclass and Sex<jupyter_code>pd.crosstab([train.Embarked, train.Pclass],[train.Sex,train.Survived],margins=True).style.background_gradient(cmap='summer_r')
plt.figure(figsize=(30,6))
sns.catplot(x='Pclass',y='Survived',hue='Sex',col='Embarked',data=train, kind="point")
plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>* We can see that men from all ports S, C, Q who were alone and belongeed to third class had the maximum chance of dying.* Rate of survival of women belonging to Pclass 1 and Pclass 2 is 1 regardless of the port they embarked from<jupyter_code>train.head()<jupyter_output><empty_output><jupyter_text>Relationship between Fare_Bin, Pclass and Survived<jupyter_code>pd.crosstab([train.Survived, train.Pclass], train.Fare_Bin, margins=True).style.background_gradient(cmap='Blues')
fare_dict<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1465147926011794, 'fraction_numerical': 0.01315457890227307, 'mean_word_length': 6.708624708624709, 'pattern_counts': {'":': 0, '<': 148, '<?xml version=': 0, '>': 150, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '37', 'dir': 'jupyter-structured-clean-dedup', 'id': '210276', 'license': 'MIT', 'n_tokens_mistral': 7113, 'n_tokens_neox': 6672, 'n_words': 1777, 'path': 'EDA/Titanic_Dataset/EDA_on_Titanic_Dataset.ipynb', 'repo_name': 'juanvera11127/AlgoCode', 'size': '19841'}