<jupyter_start><jupyter_text>TITANIC DISASTER PROBLEM FROM KAGGLE    A personal solution to this problem.        PROJECT WORK FOR ALGORITHM DESIGN - CSE 3131<jupyter_code># Importing the necessary libraries
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
# Getting the training and test set
train = pd.read_csv('titanic/train.csv')
test = pd.read_csv('titanic/test.csv')
# Taking a look at the training set
train.head()
# Getting some detailed info about the training set
train.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 12 columns):
PassengerId    891 non-null int64
Survived       891 non-null int64
Pclass         891 non-null int64
Name           891 non-null object
Sex            891 non-null object
Age            714 non-null float64
SibSp          891 non-null int64
Parch          891 non-null int64
Ticket         891 non-null object
Fare           891 non-null float64
Cabin          204 non-null object
Embarked       889 non-null object
dtypes: float64(2), int64(5), object(5)
memory usage: 83.6+ KB<jupyter_text>We can see that the 'Age' column, 'Cabin' column and 'Embarked' column have some missing values. ---  Visual Analysis of some feature columns<jupyter_code>sns.factorplot(x = 'Survived', col = 'Sex', kind = 'count', data = train)
plt.show()<jupyter_output><empty_output><jupyter_text>Clearly females have a higher likelihood of survival<jupyter_code># Calculating the percentage of female survival
train[train.Sex == 'female'].Survived
# Since length is 314, we have to divide the number of survived females by 314
females_survived = train[train.Sex == 'female'].Survived.sum()/314
print(round(females_survived * 100))
# Calculating the percentage of males survived
train[train.Sex == 'male'].Survived
# Since length is 577, we have to divide the number of survived females by 577
males_survived = train[train.Sex == 'male'].Survived.sum()
print(round(males_survived/577 * 100))<jupyter_output>19<jupyter_text>Approximately 74% females and 19% males survived. Male survival percentage is much lower than females!<jupyter_code># Using seaborn to build bar plots of the feature 'Survived' over 'Pclass'
# Gives us an idea of survival rates of 'Pclass' feature column

sns.factorplot(x = 'Survived', col = 'Pclass', kind = 'count', data = train)
plt.show()<jupyter_output><empty_output><jupyter_text>We can see that the passengers on the first class were the most likely to survive.<jupyter_code># Using seaborn to build bar plots of the feature 'Survived' over 'Embarked'
# Gives us an idea of survival rates of 'Pclass' feature column
sns.factorplot(x = 'Survived', col = 'Embarked', kind = 'count', data = train);
plt.show()
# Using seaborn to build bar plots of the feature 'Survived' over 'Fare'
sns.factorplot(x = 'Survived', col = 'Fare', kind = 'count', data = train)
plt.show()<jupyter_output><empty_output><jupyter_text>We can see that the image size is too large. We therefore have to use another method to visualize the 'Fare' feature.<jupyter_code># Plotting a histogram of 'Fare' column from the training set
sns.distplot(train.Fare, kde = False)
plt.show()
# Plotting 'Fare' and 'Survived' together to get an idea of how survival is related to the fare value.
train.groupby('Survived').Fare.hist(alpha=0.5)
plt.show()<jupyter_output><empty_output><jupyter_text>Those who paid more, had higher chances of survival! ---  Imputing missing values and creating my Machine Learning model ---  A final check on the 'Age' feature column. Obviously 'Age' feature column has an impact on the survival. But, it has got some missing values. I can fill those with either the mean of the median of the same column. But before doing that, I need to check the distribution of the 'Age' feature column in order to decide which one would be suitable.<jupyter_code># Using seaborn to plot a histogram of 'Age' column and check the distribution of the values.

sns.distplot(train.dropna().Age)    # Missing values have to be dropped else error will pop out.
plt.show()<jupyter_output><empty_output><jupyter_text>We can see that the 'Age' feature column is clearly right skewed. So mean won't be an appropriate choice. Hence, I shall fill the missing values with the median of the column. ---<jupyter_code># Storing the response data for later use
train_survived = train.Survived

# Concatenating the training and the test sets to impute the missing values and work on a combined data
train_test_data = pd.concat([train.drop(['Survived'], axis = 1), test])

# Checking the info of the new concatenated data
train_test_data.info()
# Filling the 'Age' and 'Fare' feature columns with the median values

train_test_data['Age'] = train_test_data.Age.fillna(train_test_data.Age.median())
train_test_data['Fare'] = train_test_data.Fare.fillna(train_test_data.Fare.median())

# Checking the info again to see if the missing values have been filled up.
train_test_data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 1309 entries, 0 to 417
Data columns (total 11 columns):
PassengerId    1309 non-null int64
Pclass         1309 non-null int64
Name           1309 non-null object
Sex            1309 non-null object
Age            1309 non-null float64
SibSp          1309 non-null int64
Parch          1309 non-null int64
Ticket         1309 non-null object
Fare           1309 non-null float64
Cabin          295 non-null object
Embarked       1307 non-null object
dtypes: float64(2), int64(4), object(5)
memory usage: 122.7+ KB<jupyter_text>The missing values of 'Age' and 'Fare' columns have now been filled up. Now we just have to take care of the 'Cabin' and 'Embarked' feature columns.Since, the 'Cabin' feature column has got only 295 filled in columns, I am gonna ignore that column for this model.<jupyter_code># Removing the 'Cabin' feature column
train_test_data.drop(['Cabin'], axis = 1, inplace = True)

train_test_data.info()
# Filling the 'Embarked' feature column with the most frequent value

train_test_data = train_test_data.fillna(train_test_data['Embarked'].value_counts().index[0])

train_test_data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 1309 entries, 0 to 417
Data columns (total 10 columns):
PassengerId    1309 non-null int64
Pclass         1309 non-null int64
Name           1309 non-null object
Sex            1309 non-null object
Age            1309 non-null float64
SibSp          1309 non-null int64
Parch          1309 non-null int64
Ticket         1309 non-null object
Fare           1309 non-null float64
Embarked       1309 non-null object
dtypes: float64(2), int64(4), object(4)
memory usage: 112.5+ KB<jupyter_text>ALL THE THINGS HAVE BEEN TAKEN CARE OF NOW!<jupyter_code># Now we encode the data by making dummy variables of the required feature columns as Categorical values might not be 
# accepted by the machine learning model

# 'Embarked' and 'Sex' are the two columns am focusing on.

train_test_data = pd.get_dummies(train_test_data, columns = ['Sex'], drop_first=True)
train_test_data = pd.get_dummies(train_test_data, columns = ['Embarked'], drop_first=True)

# Dropping one column from each of these as we might get caught in the 'DUMMY VARIABLE TRAP'. So to avoid that,
# we have to take n - 1 columns.

train_test_data.describe()<jupyter_output><empty_output><jupyter_text>We can see that the 'Sex' feature column and the 'Embarked' feature column have been divided into 2 and 3 separate dummy variable column each and have been sliced off of one column each to avoid the DUMMY VARIABLE TRAP.Using get_dummies() which is a pandas function, the 'Sex' column was divided into 'Sex_Male' and 'Sex_Female' and the later was discarded. Similarly, 'Embarked'column was divided into 'Embarked_Q', 'Embarked_S' and 'Embarked_C' and 'Embarked_C' was sliced off.<jupyter_code># Now, I am selecting the columns on which I want to build my Machine Learning model.
# Since am a beginner, I thought PassengerId might not have much effect on the prediction, althought i might be wrong.

train_test_data = train_test_data[['Pclass', 'Age', 'SibSp', 'Parch', 'Fare', 'Sex_male', 'Embarked_Q', 'Embarked_S']]

# Checking if all is good!
train_test_data.head()<jupyter_output><empty_output><jupyter_text>All good!<jupyter_code># Final check on the info and getting the description

train_test_data.info()
train_test_data.describe()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 1309 entries, 0 to 417
Data columns (total 8 columns):
Pclass        1309 non-null int64
Age           1309 non-null float64
SibSp         1309 non-null int64
Parch         1309 non-null int64
Fare          1309 non-null float64
Sex_male      1309 non-null uint8
Embarked_Q    1309 non-null uint8
Embarked_S    1309 non-null uint8
dtypes: float64(2), int64(3), uint8(3)
memory usage: 65.2 KB<jupyter_text>---  Building the 'Support Vector Machine Classifier.<jupyter_code># Splitting the dataset back into the training and test sets and then using scikit learn library to convert them into arrays
X = train_test_data.iloc[:891].values
y = train_survived.values
X_test = train_test_data.iloc[891:].values

X, y, X_test
# Importing the classifier and fitting it to the training set

from sklearn.svm import SVC
classifier = SVC(kernel = 'linear', random_state = 0)
classifier.fit(X, y)
# Predicting the test set results and then formatting the test set to hold only the required two columns.
y_pred = classifier.predict(X_test)

test['Survived'] = y_pred
test = test[['PassengerId', 'Survived']]
# Converting the result to a csv file.
test.to_csv('titanic_submission.csv', index = False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08380462724935733, 'fraction_numerical': 0.03506426735218509, 'mean_word_length': 4.418384401114206, 'pattern_counts': {'":': 0, '<': 52, '<?xml version=': 0, '>': 52, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '540234', 'license': 'MIT', 'n_tokens_mistral': 3338, 'n_tokens_neox': 3037, 'n_words': 1165, 'path': 'TITANIC DISASTER PROBLEM.ipynb', 'repo_name': 'SibasishTripathyy/Titanic-kaggle-project', 'size': '9725'}