<jupyter_start><jupyter_text>Case study on Data Preprocessing**<NAME>***NORKA B5* **Do the following on titanic dataset.**1. Load the dataset into python environment2. Make ‘PassengerId’ as the index column3. Check the basic details of the dataset4. Fill in all the missing values present in all the columns in the dataset5. Check and handle outliers in at least 3 columns in the dataset6. Do min max scaling on the feature set (Take ‘Survived’ as target)<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>**1. Load the dataset into python environment****2. Make ‘PassengerId’ as the index column**<jupyter_code>df = pd.read_csv('./datasets/titanic_dataset.csv', index_col='PassengerId')
df.head()<jupyter_output><empty_output><jupyter_text>*** **3. Check the basic details of the dataset**<jupyter_code>df.columns
print('Shape of titanic dataset is :: ',df.shape)
print('Basic summary of titanic dataset :: \n')
df.info()<jupyter_output>Basic summary of titanic dataset :: 

<class 'pandas.core.frame.DataFrame'>
Int64Index: 891 entries, 1 to 891
Data columns (total 11 columns):
 #   Column    Non-Null Count  Dtype  
---  ------    --------------  -----  
 0   Survived  891 non-null    int64  
 1   Pclass    891 non-null    int64  
 2   Name      891 non-null    object 
 3   Sex       891 non-null    object 
 4   Age       714 non-null    float64
 5   SibSp     891 non-null    int64  
 6   Parch     891 non-null    int64  
 7   Ticket    891 non-null    object 
 8   Fare      891 non-null    float64
 9   Cabin     204 non-null    object 
 10  Embarked  889 non-null    object 
dtypes: float64(2), int64(4), object(5)
memory usage: 83.5+ KB<jupyter_text>From the stats it is clear that not all columns are fully filled.<jupyter_code>df.groupby('Sex').count()
df.nunique().sort_values(ascending=True)<jupyter_output><empty_output><jupyter_text>6 Columns in this dataset describes some category with a max for 7 unique values for Parch and SibSp<jupyter_code>df.describe().round(0)<jupyter_output><empty_output><jupyter_text>4. Fill in all the missing values present in all the columns in the dataset<jupyter_code>df.isna().sum()
print('Number of rows containing NAN = ', df.isna().any(axis = 1).sum())
df['Age'].fillna(round(df['Age'].mean(), 0), inplace = True)
df['Cabin'].fillna(df['Cabin'].mode().loc[0], inplace = True)
df['Embarked'].fillna(df['Embarked'].mode().loc[0], inplace = True)
df.isna().sum()<jupyter_output><empty_output><jupyter_text>5. Check and handle outliers in at least 3 columns in the dataset<jupyter_code>def find_outliers(series):
    Q1 = np.percentile(series, 25, interpolation = 'midpoint')
    Q2 = np.percentile(series, 50, interpolation = 'midpoint')
    Q3 = np.percentile(series, 75, interpolation = 'midpoint')
    IQR = Q3 - Q1
    low_lim = Q1 - 1.5*IQR
    up_lim = Q3 + 1.5*IQR
    print(f'Q1 = {Q1}, Q2 = {Q2} and Q3 = {Q3}')
    print(f'IQR = {IQR}, Upper limit = {up_lim} and Lower limit = {low_lim}')
    outlier_list = []
    for x in series:
        if((x>up_lim) or (x<low_lim)):
            outlier_list.append(x)
    print(f'There are {len(outlier_list)} Outliers for {series.name}, they are {outlier_list}')
    return outlier_list
df['Fare'].plot.box()
plt.show()
age_outliers = find_outliers(df['Age'])
age_index = df[df['Age'].isin(age_outliers)].index
age_index
df.drop(age_index, inplace = True)
df.shape
fare_outliers = find_outliers(df['Fare'])
fare_index = df[df['Fare'].isin(fare_outliers)].index
fare_index
df.drop(fare_index, inplace = True)
df.shape
sib_outliers = find_outliers(df['SibSp'])
sib_index = df[df['SibSp'].isin(sib_outliers)].index
sib_index
df.drop(sib_index, inplace = True)
df.shape<jupyter_output><empty_output><jupyter_text>6. Do min max scaling on the feature set (Take ‘Survived’ as target)<jupyter_code>X = df[['Pclass', 'Age', 'SibSp','Parch','Fare']]
X.describe()
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler(feature_range =(0,1))
X = scaler.fit_transform(X)
X= pd.DataFrame(X)
X.describe()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13341433778857836, 'fraction_numerical': 0.030376670716889428, 'mean_word_length': 4.3803921568627455, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '341229', 'license': 'MIT', 'n_tokens_mistral': 1570, 'n_tokens_neox': 1439, 'n_words': 393, 'path': 'notebooks/cs-06-preprocessing.ipynb', 'repo_name': 'sreeh4ri/ictk_projects', 'size': '4121'}