<jupyter_start><jupyter_text>***Data Cleansing ON TITANIC DATASET***<jupyter_code># Importing Libraries
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
# Loading the data
df = pd.read_csv('/kaggle/input/titanic/train.csv')
df.head()
df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 891 entries, 0 to 890
Data columns (total 12 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   PassengerId  891 non-null    int64  
 1   Survived     891 non-null    int64  
 2   Pclass       891 non-null    int64  
 3   Name         891 non-null    object 
 4   Sex          891 non-null    object 
 5   Age          714 non-null    float64
 6   SibSp        891 non-null    int64  
 7   Parch        891 non-null    int64  
 8   Ticket       891 non-null    object 
 9   Fare         891 non-null    float64
 10  Cabin        204 non-null    object 
 11  Embarked     889 non-null    object 
dtypes: float64(2), int64(5), object(5)
memory usage: 83.7+ KB<jupyter_text>Just by looking at the data I can tell that I have some missing data in my *Age*, *Embarked* and *Cabin* column. Lets see how much of the data is actually missing<jupyter_code>pd.DataFrame(df.isna().sum().sort_values(ascending=False)/len(df)*100)<jupyter_output><empty_output><jupyter_text>Now I can tell that in column - Cabin, Age and Embarked we have 77.10%, 19.86% and 0.22% of missing data respectively. ***brief description of each column***    PassengerId -- passenger identifiation unique ID    Survived -- Survivor flag (0 = dead, 1 = survived)    Pclass -- ticket class:        1 = 1st Class        2 = 2nd Class        3 = 3rd Class    Name -- as is    Sex -- as is    Age -- as is    SibSp -- number of siblings/spouses on Titanic    Parch -- number of parents/children on Titanic    Ticket -- ticket number    Fare -- price    Cabin -- Cabin number    Embarked -- port of departure        S = Southampton        C = Cherbourg        Q = Queenstown<jupyter_code># Number of people survived vs not servived
plt.figure(figsize=(16,8))
df["Survived"].value_counts().plot.bar()
plt.show
non_survived_passenger = round(df["Survived"].value_counts()[0]/len(df)*100,3)
survived_passenger = round(df["Survived"].value_counts()[1]/len(df)*100,3)
print(f"Percentage of passengers survived: {survived_passenger}%")
print(f"Percentage of passengers who could not survived: {non_survived_passenger}%")
#Maximum and Minimun age of the passengers
print(f"Maximum age is: {df['Age'].max()}")
print(f"Minimum age is: {df['Age'].min()}")
#Univariate Analysis
df.plot(kind="scatter",x="Age",y="Survived")
plt.title("AGE VS SURVIVED")
plt.show()<jupyter_output><empty_output><jupyter_text>I did not see any correlations between Age and Survived columns<jupyter_code>df.plot(kind="scatter", x="Fare", y="Survived")
plt.title("FARE VS SURVIVED")
plt.show()
plt.figure(figsize=(16,8))
plt.pie(df["SibSp"].value_counts().values, autopct="%1.0f%%", labels=[0,1,2,3,4,5,6])
plt.show()
df['Parch'].value_counts().plot.bar()
plt.figure(figsize=(16,8))
sns.histplot(df['Fare'])
plt.show()
plt.figure(figsize=(16,8))
df["Sex"].value_counts().plot.bar()
plt.show()
plt.figure(figsize=(16,8))
df["Embarked"].value_counts().plot.bar()
plt.show()<jupyter_output><empty_output><jupyter_text>***Preparing the dataset for machine learning***Seprating numerical and categorical values. Performing preprocessing operations.<jupyter_code># Convert the parent children and sibling spouse columns into a float ones 
# because they don't show a categorial value.
df['Parch'] = df['Parch'].astype(float)
df['SibSp'] = df['SibSp'].astype(float)
features = df.drop("Survived", axis=1)
labels = df["Survived"].values
num_cols = [col for col in features.columns if features[col].dtype in [float]]
print(num_cols)
# Checking for the cardinal value which should be less than 10
cat_cols = [col for col in features.columns if col not in num_cols and df[col].unique().shape[0]<10]
print(cat_cols)
df[cat_cols].head()
df[num_cols].head()<jupyter_output><empty_output><jupyter_text>Columns with missing values are-1. Age1. Cabin1. Embarked<jupyter_code># Imputating numerical columns
 # Taking the average value for Age column
df['Age'].mean()
features['Age'].mean()
features[num_cols]
# Option 1
# using fill na
features['Age'].fillna(features['Age'].mean()).values
features['Age'].fillna(features['Age'].mean()).isna().sum()
# option 2 (Prefered)
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
imputer = SimpleImputer(strategy="mean")
imputer.fit_transform(features[num_cols])
# Making a numerical data pipeline
from sklearn.preprocessing import StandardScaler
num_pipeline = Pipeline([
    ("imputer",SimpleImputer(strategy='mean')),
    ("scaler",StandardScaler())
])
num_pipeline.fit_transform(features[num_cols])<jupyter_output><empty_output><jupyter_text>So, We have prepared the numerical data. Lets move towards the categorical dataNote- There are no library for imputating categorical data<jupyter_code># Lets again observe the categorical data
features[cat_cols].head()
features[cat_cols].isna().sum()<jupyter_output><empty_output><jupyter_text>Only two values are missing from the column "Embarked"<jupyter_code>features["Embarked"].value_counts()
# From above we can see that the most frequent value here is "S".
# So let's fill that value in our 2 missing "Embarked" rows
features["Embarked"].fillna("S").isna().sum()
# Creating a custom imputer
from sklearn.base import BaseEstimator, TransformerMixin
class MostFrequentImputer(BaseEstimator, TransformerMixin):
    def fit(self, X, y=None):
        self.most_frequent = pd.Series([X[c].value_counts().index[0] for c in X], index=X.columns)
        return self
    def transform(self,X,y=None):
        return X.fillna(self.most_frequent)
# One hot encoding
from sklearn.preprocessing import OneHotEncoder
cat_pipeline = Pipeline([
    ('cat_imputer',MostFrequentImputer()),
    #('one_hot',OneHotEncoder(sparse=False))
    ('one_hot',OneHotEncoder())
])
cat_pipeline.fit_transform(features[cat_cols])
X_num = num_pipeline.fit_transform(features[num_cols])
X_cat = cat_pipeline.fit_transform(features[cat_cols]).toarray()
X_final = np.c_[X_num,X_cat]
print(X_final[:5])
X_final.shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13044159098026933, 'fraction_numerical': 0.02395865956780457, 'mean_word_length': 4.702678571428572, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '23488', 'license': 'CC0-1.0', 'n_tokens_mistral': 2266, 'n_tokens_neox': 2123, 'n_words': 561, 'path': 'Day3/titanic-dataset-data-cleansing.ipynb', 'repo_name': 'Maaz199/LetsUpgrage-Assignment', 'size': '6386'}