<jupyter_start><jupyter_text>We have already read the data and now its time for data preprocessing.Looking at the various mean values and also finding if there is any null value in our dataset and various other steps .<jupyter_code>train.describe()
train.isnull()<jupyter_output><empty_output><jupyter_text>From here we can see that there are null values in our Cabin and Age column but a better way to see this is by plotting heatmap using seaborn lib or by just getting info about the datset.<jupyter_code>train.info()
plt.figure(figsize=(10,10))
sns.heatmap(train.isnull(),yticklabels=False,cbar=True)<jupyter_output><empty_output><jupyter_text>From here we can see that there are null values in **Embarked column** as well and we might want to remove this null values for a better result. So lets deal with the Age dataset first and try to replace these NaN values with some values that can do well.One way to do so is to just replace the values with the mean of the ages of all but that will not be a good option as you will see further how the value changes. Lets plot some of the graphs and identify the relationship of age with other features.<jupyter_code>plt.figure(figsize=(15,10))
sns.boxplot('Pclass','Age',data=train)<jupyter_output><empty_output><jupyter_text>From this we can see that the average age for Pclass 1 is around 38 where as for other classes its 29 and 23 respectively. So just by doing the mean we would have replaced the value to 29.6 which would have not performed as good as this . So lets replace the NaN values with the values we found for each class.<jupyter_code>def Age_1(cols) :
    Age=cols[0]
    Pclass=cols[1] 
    
    if pd.isnull(Age) :
        if Pclass==1 :
            return 38
        elif Pclass==2:
            return 29
        else :
            return 24
    else :
        return Age
                
train['Age']=train[['Age','Pclass']].apply(Age_1,axis=1)<jupyter_output><empty_output><jupyter_text>Here we have replaced the values of Age column with their respective mean values.Now by seeing the data we can see that Passengerid, Cabin , Ticket, and Name is of no use for determining whether he survived or not .So lets just remove them from our train as well as test dataset.<jupyter_code>train.drop(['PassengerId','Ticket','Cabin','Name'],inplace=True,axis=1)
X_test.drop(['PassengerId','Ticket','Cabin','Name'],inplace=True,axis=1)<jupyter_output><empty_output><jupyter_text>Now there is NaN value in Embarked column and that also only two of them . So Just dropping these two values wont make much difference in our dataset.<jupyter_code>train=train.dropna(axis=0,how='any')<jupyter_output><empty_output><jupyter_text>Lets see if our dataset contains any NaN values or not .<jupyter_code>plt.figure(figsize=(10,10))
sns.heatmap(train.isnull(),yticklabels=False,cbar=True)<jupyter_output><empty_output><jupyter_text>Here we will also have to remove the NaN values and replace it will other values and repeat the same steps on your X_test dataset also bcz it also contains null values. I have just done that in one cell but you can do it by following previous steps .<jupyter_code>plt.figure(figsize=(15,10))
sns.boxplot('Pclass','Age',data=X_test)

def Age_1(cols) :
    Age=cols[0]
    Pclass=cols[1] 
    
    if pd.isnull(Age) :
        if Pclass==1 :
            return 42 
        elif Pclass==2:
            return 25
        else :
            return 22
    else :
        return Age
                
X_test['Age']=X_test[['Age','Pclass']].apply(Age_1,axis=1)<jupyter_output><empty_output><jupyter_text>Now   Lets start visualising it in depth and get some insight out of it .<jupyter_code>sns.countplot('Survived',hue='Sex',data=train)<jupyter_output><empty_output><jupyter_text>This clearly shows that a lot of females survived over males .<jupyter_code>plt.figure(figsize=(15,10))
sns.boxplot('Pclass','Fare',data=train)<jupyter_output><empty_output><jupyter_text>This tells us that for Pclass 1 they actually paid higher money while other classes didnt pay much money but also Pclass 1 and 3 have a lotmore outliers that would led to overfitting so it must be taken care off.<jupyter_code>sns.countplot('Survived',hue='Pclass',data=train)<jupyter_output><empty_output><jupyter_text>Survival rate of Pclass is much more than other but for Pclass 2 is almost the same and so by seeing this we cant just ignore the Fare feature and drop it and it is very crucial for further predictions.<jupyter_code>plt.figure(figsize=(15,5))
sns.distplot(train['Fare'],bins=40)<jupyter_output><empty_output><jupyter_text>This doesnt look like a proper gaussian curve and might tend to overfit the curve as it has very less values of people survived for fare greater than 80$.<jupyter_code>train_fare = train['Fare'].map(lambda i: np.log(i) if i > 0 else 0)
plt.figure(figsize=(15,5))
sns.distplot(train_fare,bins=40)<jupyter_output><empty_output><jupyter_text>Now we can see that the curve it equally distributed and will perfrom well under any classifier.<jupyter_code>fare=train['Fare'].map(lambda i: np.log(i) if i > 0 else 0)<jupyter_output><empty_output><jupyter_text>Replacing the Fare values by their logerithmic values Now lets replace the Embarked and Sex columns with their binary values<jupyter_code>sex=pd.get_dummies(train['Sex'],drop_first=True)
emb=pd.get_dummies(train['Embarked'],drop_first=True)
y_train=train['Survived']
train.drop(['Embarked','Sex','Survived','Fare'],inplace=True,axis=1)<jupyter_output><empty_output><jupyter_text>The dummies are created and now the original dataset we need to concate with the newly created one<jupyter_code>X_train=pd.concat([train,sex,emb,fare],axis=1)
y_test.drop('PassengerId',inplace=True,axis=1)
X_train<jupyter_output><empty_output><jupyter_text>This is our final dataset for training the dataset and same goes for y_train .I have to perform the following steps on X_test i.e. test data as well bcz even it contains some of the data like this .Repeat the steps<jupyter_code>test_fare=X_test['Fare'].map(lambda i: np.log(i) if i > 0 else 0)
test_sex=pd.get_dummies(X_test['Sex'],drop_first=True)
test_emb=pd.get_dummies(X_test['Embarked'],drop_first=True)
X_test.drop(['Embarked','Sex','Fare'],inplace=True,axis=1)
X_test=pd.concat([X_test,test_sex,test_emb,test_fare],axis=1)
Y_test=pd.concat([X_test,test_sex,test_emb,test_fare,y_test],axis=1)
Y_test=Y_test.dropna(axis=0,how='any')

y_test=Y_test['Survived']<jupyter_output><empty_output><jupyter_text>Now we only problm we would face is that our X_test contains 417 rows of data whereas our y_test contains 418 rows of data . So instead initiall while removing the NaN value from fare dataset we would have just used OneHotEncoder for categorical data and IterativeImputer for Numerical data to solve it . But it just one value other way is to just concate it and then drop that value ,  so that it will remove it from our y_test dataset as well. Now our dataset preprocessing and visualisation is complete and now its time to use the proper method to train and test the dataset to get the best output.  Using Logistic Regression Algorithm<jupyter_code>from sklearn.linear_model import LogisticRegression

from sklearn import metrics 
from sklearn.metrics import confusion_matrix 
from sklearn.metrics import accuracy_score 
from sklearn.metrics import classification_report 
lgr=LogisticRegression(max_iter=500)
lgr_train=lgr.fit(X_train,y_train)
y_pred = lgr.predict(X_test)
print(accuracy_score(y_test,y_pred))
print(classification_report(y_test,y_pred))
print(confusion_matrix(y_test,y_pred))
from sklearn.ensemble import RandomForestClassifier
rnd_clf=RandomForestClassifier(n_estimators=1000,max_leaf_nodes=16,n_jobs=-1)
rnd_clf.fit(X_train,y_train)
y_pred=rnd_clf.predict(X_test)
print(accuracy_score(y_test,y_pred))
print(classification_report(y_test,y_pred))
print(confusion_matrix(y_test,y_pred))<jupyter_output>0.9019138755980861
              precision    recall  f1-score   support

           0       0.89      0.97      0.93       266
           1       0.94      0.78      0.85       152

    accuracy                           0.90       418
   macro avg       0.91      0.88      0.89       418
weighted avg       0.90      0.90      0.90       418

[[258   8]
 [ 33 119]]<jupyter_text>Using Bagging technique Also used GridSearch to use proper hyperparameter and results are much better than using just the logistic regression algorithm.<jupyter_code>from sklearn.ensemble import BaggingClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.tree import DecisionTreeClassifier

params = [{'max_leaf_nodes': list(range(2, 100)), 'min_samples_split': [2, 3, 4]}]
b_clf=BaggingClassifier(GridSearchCV(DecisionTreeClassifier(random_state=42),params,cv=3,verbose=1),n_estimators=1000,max_samples=100,bootstrap=True,n_jobs=-1)
b_clf.fit(X_train,y_train)
y_pred=b_clf.predict(X_test)
print(accuracy_score(y_test,y_pred))
print(classification_report(y_test,y_pred))
print(confusion_matrix(y_test,y_pred))<jupyter_output>0.9952153110047847
              precision    recall  f1-score   support

           0       0.99      1.00      1.00       266
           1       1.00      0.99      0.99       152

    accuracy                           1.00       418
   macro avg       1.00      0.99      0.99       418
weighted avg       1.00      1.00      1.00       418

[[266   0]
 [  2 150]]
{'fraction_non_alphanumeric': 0.08732694355697551, 'fraction_numerical': 0.030670926517571886, 'mean_word_length': 4.636854741896759, 'pattern_counts': {'":': 0, '<': 75, '<?xml version=': 0, '>': 78, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '29634', 'license': 'Apache-2.0', 'n_tokens_mistral': 3218, 'n_tokens_neox': 2993, 'n_words': 939, 'path': 'titanic-dataset-accuracy-99%.ipynb', 'repo_name': 'Sushan570/titanic-survival-', 'size': '9390'}