<jupyter_start><jupyter_text>IMPORT LIBRARIES<jupyter_code>import numpy as np 
import pandas as pd
import numpy as np

from sklearn.compose import ColumnTransformer
from sklearn.datasets import fetch_openml
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.decomposition import PCA
from sklearn.impute import SimpleImputer, KNNImputer
from sklearn.preprocessing import RobustScaler, OneHotEncoder
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split, cross_val_score, RandomizedSearchCV
import seaborn as sns
import matplotlib.pyplot as plt
import missingno as msno<jupyter_output><empty_output><jupyter_text>READ THE DATA:- In this challenge you will use the dataset Titanic disaster.- With the following lines of code, you can import the dataset form the file "train.csv", extract te column of labels (column "Survived") and read the infos of the other features.---<jupyter_code># Read the dataset
features, labels = fetch_openml("titanic", version=1, as_frame=True, return_X_y=True)

# Split the dataset and analyse only the training set to avoid biases
X_train, X_test, y_train, y_test = \
    train_test_split(features, labels, stratify=labels, test_size=0.2, random_state=42)<jupyter_output><empty_output><jupyter_text>Visualize the first rows<jupyter_code>X_train.head(2)<jupyter_output><empty_output><jupyter_text>Check if there are missing values in the dataframe<jupyter_code>X_train.isnull().any()<jupyter_output><empty_output><jupyter_text>We can see that *age, fare, cabin, boat, body, home.dest* have missing values. Let's determine how many<jupyter_code>(X_train.isnull().sum()/X_train.shape[0])*100
msno.matrix(X_train)
plt.show()<jupyter_output><empty_output><jupyter_text>We can see that cabin, body, boat, are very corrupted (78%, 62%, 91% missing respectively) so it may be wise to just ignore those features. The home.dest columns is also very corrupted and it's practically not possible to imputate those values based on the data from the other passagers and from the other columns, so it will be also ignored.Also from here we see that for the age and fare features wee will need to perform imputation.<jupyter_code>X_train.drop(columns=['cabin', 'boat', 'body', 'home.dest'], inplace=True)
X_test.drop(columns=['cabin', 'boat', 'body', 'home.dest'], inplace=True)<jupyter_output><empty_output><jupyter_text>For the remaining columns lets go in detail<jupyter_code>X_train.head(2)<jupyter_output><empty_output><jupyter_text>Name<jupyter_code>X_train.name.sample(10)<jupyter_output><empty_output><jupyter_text>Is interesting to see that in the name there is information of the marital/social status encoded in teh Mr. Mrs. Master. etc. That could be useful Class<jupyter_code>X_train.pclass.unique()
X_train['pclass'] = X_train.pclass.astype(int)
X_test['pclass'] = X_test.pclass.astype(int)<jupyter_output><empty_output><jupyter_text>Sex<jupyter_code>X_train.sex.value_counts()<jupyter_output><empty_output><jupyter_text>We can see an unbalanced dataset with more men than female Age<jupyter_code>plt.figure(figsize=(15,1))
sns.boxplot(x=X_train.age)
sns.despine()
plt.show()<jupyter_output><empty_output><jupyter_text>We can see no impossible/wrongly entered values in age (age100) Sibsp<jupyter_code>X_train.sibsp.value_counts()<jupyter_output><empty_output><jupyter_text>Parch<jupyter_code>X_train.parch.value_counts()<jupyter_output><empty_output><jupyter_text>Ticket<jupyter_code>X_train.ticket.dtype<jupyter_output><empty_output><jupyter_text>The ticket column don't seem to have useful information. It will be dropped.<jupyter_code>X_train.drop(columns=['ticket'], inplace=True)
X_test.drop(columns=['ticket'], inplace=True)<jupyter_output><empty_output><jupyter_text>Fare<jupyter_code>plt.figure(figsize=(15,2))
sns.histplot(data=X_train, x='fare')
sns.despine()
plt.show()
plt.figure(figsize=(15,2))
sns.boxplot(data=X_train, x='fare')
sns.despine()
plt.show()<jupyter_output><empty_output><jupyter_text>Here we can see that maybe the fare over 500 is a mistake, our someone got scammed XD Embarked<jupyter_code>X_train.embarked.value_counts()<jupyter_output><empty_output><jupyter_text>Lets analyse the correlation between features<jupyter_code>X_comb = pd.concat([X_train, y_train.astype(float)], axis=1)
g = sns.heatmap(
    X_comb[['pclass', 'sex', 'age', 'sibsp', 'parch', 'fare', 'survived']].corr(),
    annot=True
)<jupyter_output><empty_output><jupyter_text>From here we can get some important elements:- The Class of the passenger is inversely correlated with the survival, passengers in the third class had less changes to survive.- The Age is not correlated with survival- Sibsp and Parch, both features indicating familiar status of the passager are correlated.- The fare is positively correlated with the survival rate as (sadly) expected.  Feature Engineering - Evaluate if there are features with missing values and modify them (see https://scikit-learn.org/stable/modules/impute.html)- Find meaningful features (some features are not needed because do not contain useful infos)- Encode categorical features with numerical values (see https://scikit-learn.org/stable/modules/preprocessing.htmlencoding-categorical-features) Lets get the title out of the name and combine parch and sibs in a family feature. Since the new family feature will be highly correlated with the other two and also contain them,  parch and sibl will be dropped. In the case of the name, we will retain just the title.<jupyter_code>for dataset in [X_train, X_test]:
    dataset['family_size'] = dataset['parch'] + dataset['sibsp']
    dataset.drop(['parch', 'sibsp'], axis=1, inplace=True)
    dataset['is_alone'] = True
    dataset.loc[dataset['family_size'] > 1, 'is_alone'] = False
    dataset['title'] =  dataset['name'].str.split(", ", expand=True)[1].str.split(".", expand=True)[0]
    dataset.drop(["name"], axis=1, inplace=True)
X_train.head()
X_train.title.value_counts()<jupyter_output><empty_output><jupyter_text>There are many titles,lets group them.<jupyter_code>pd.crosstab(X_train['title'], X_train['sex'])
print(f"Miss: {np.sum(y_train.astype(int)[X_train.title == 'Miss']) / len(X_train.title == 'Miss')}")
print(f"Mrs: {np.sum(y_train.astype(int)[X_train.title == 'Mrs']) / len(X_train.title == 'Mrs')}")<jupyter_output>Miss: 0.13371537726838587
Mrs: 0.1174785100286533<jupyter_text>The difference between Miss and Mrs seems to be insignificant, group them together in one. Lets also group all the rare titles.<jupyter_code>X_comb = pd.concat([X_train, X_test])
rare_titles = (X_comb['title'].value_counts() < 10)
rare_titles
for dataset in [X_train, X_test]:
    dataset.loc[dataset.title == 'Miss', 'title'] = 'Mrs'
    dataset['title'] = dataset.title.apply(lambda x: 'rare' if rare_titles[x] else x)<jupyter_output><empty_output><jupyter_text>Imputation We can work separetely categorical and cuantitative features Categorical:<jupyter_code>def get_cat_cols(df):
    obj_cols = df.columns[df.dtypes == 'object']
    cat_cols = df.columns[df.dtypes == 'category']
    return set(obj_cols).union(set(cat_cols))

cat_cols = get_cat_cols(X_train)
cat_cols<jupyter_output><empty_output><jupyter_text>Even if in the training data we didn't find any missing value for the categorical variables in the test set they might be, so we consider the case in the transformer pipeline, using "most frequent strategy". We are going to transform the categorical variables into numeric ones to enlarge the spectrum of models we can use. After using one hot encoding, PCA transformation is used in order to reduce the dimensionality of the complete categorical encodings and retain the combined features that explain the most of the variability of the data.<jupyter_code>cat_cols = ['embarked', 'sex', 'pclass', 'title', 'is_alone']
cat_transformer = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='most_frequent')),
    ('onehot', OneHotEncoder(handle_unknown='ignore', sparse=False)),
    ('pca', PCA(n_components=10))
])<jupyter_output><empty_output><jupyter_text>Numerical We will perfom the inputation with knn of 5 neighbours. Since we have seen that our numerical variables have several outliers, we can Scale and Standarize these variables considering median and iqr in order no to use biased central position estimators.<jupyter_code>num_cols = ['age', 'fare', 'family_size']
num_transformer = Pipeline(steps=[
    ('imputer', KNNImputer(n_neighbors=5)),
    ('scaler', RobustScaler())
])<jupyter_output><empty_output><jupyter_text>Combined pipeline<jupyter_code>preprocessor = ColumnTransformer(
    transformers=[
        ('num', num_transformer, num_cols),
        ('cat', cat_transformer, cat_cols)
    ]
)<jupyter_output><empty_output><jupyter_text>Model selection TRAIN AND TEST DIFFERENT CLASSIFIERS:- Find the best parameters for each classifier with a stratified 10-fold cross validation (see https://scikit-learn.org/stable/modules/cross_validation.html)- Try these models:>- Naive Bayes (see https://scikit-learn.org/stable/modules/naive_bayes.html) >- kNN (see https://scikit-learn.org/stable/modules/neighbors.htmlnearest-neighbors-classification)>- Decision tree (see https://scikit-learn.org/stable/modules/tree.html)>- Logistic regression (see https://scikit-learn.org/stable/modules/linear_model.htmllogistic-regression)- Show the final performances in terms of accuracy<jupyter_code>from sklearn.naive_bayes import GaussianNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
clf = Pipeline(steps=[('preprocessor', preprocessor),
                      ('classifier', "passthrough")])
num_transformer_dist = {'preprocessor__num__imputer__n_neighbors': list(range(2, 15)),
                        'preprocessor__num__imputer__add_indicator': [True, False]}

cat_transformer_dist = {'preprocessor__cat__imputer__strategy': ['most_frequent', 'constant'],
                        'preprocessor__cat__imputer__add_indicator': [True, False],
                        'preprocessor__cat__pca__n_components': list(range(2, 15))}

classifier_dist_1 = {
    'classifier': [GaussianNB()],
}
classifier_dist_2 = {
    'classifier': [RandomForestClassifier()],
    'classifier__n_estimators': list(range(50, 500)),
    'classifier__max_depth': list(range(2, 20)),
    'classifier__bootstrap': [True, False],
}
classifier_dist_3 = {
    'classifier': [NearestNeighbors()],
    'classifier__n_neighbors': list(range(5, 35)), # 35 ~ sqrt(len(train_set))
    'classifier__size': [1,20,40],
    'classifier__metric': ['euclidean', 'manhattan', 'chebyshev']
}
classifier_dist_4 = {
    'classifier': [DecisionTreeClassifier()],
    'classifier__criterion': ['gini', 'entropy'],
    'classifier__splitter': ['best', 'random'],
    'classifier__max_depth': list(range(5, 20))+[None],
    'classifier__max_features':['auto', 'sqrt', 'log2', None],
}

param_dist = {**num_transformer_dist,
              **cat_transformer_dist, 
              **classifier_dist_1, 
              **classifier_dist_2,
              **classifier_dist_3,
              **classifier_dist_4}

random_search = RandomizedSearchCV(
    clf,
    param_distributions=param_dist,
    scoring='accuracy', n_iter=100, cv=10
)
random_search.fit(X_train, y_train)
scores = random_search.best_estimator_.predict(X_test)
print('\nPrediction accuracy:')
print('{:.2%}\n'.format(accuracy_score(y_test, scores)))<jupyter_output>Prediction accuracy:
79.77%
{'fraction_non_alphanumeric': 0.1149922185716756, 'fraction_numerical': 0.011153380598305378, 'mean_word_length': 6.3208860759493675, 'pattern_counts': {'":': 0, '<': 104, '<?xml version=': 0, '>': 108, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '634568', 'license': 'MIT', 'n_tokens_mistral': 3926, 'n_tokens_neox': 3741, 'n_words': 952, 'path': 'challenge_2/Challenge_2.ipynb', 'repo_name': 'joaco18/maia-italy-ml-dl', 'size': '11566'}