<jupyter_start><jupyter_text>Table of Contents 1&nbsp;&nbsp;VARIABLE DESCRIPTIONS:  VARIABLE DESCRIPTIONS:* survival        Survival            (0 = No; 1 = Yes)* pclass          Passenger Class (1 = 1st; 2 = 2nd; 3 = 3rd)* name            Name* sex             Sex* age             Age* sibsp           Number of Siblings/Spouses Aboard* parch           Number of Parents/Children Aboard* ticket          Ticket Number* fare            Passenger Fare* cabin           Cabin* embarked        Port of Embarkation (C = Cherbourg; Q = Queenstown; S = Southampton)SPECIAL NOTES:Pclass is a proxy for socio-economic status (SES) 1st ~ Upper; 2nd ~ Middle; 3rd ~ LowerAge is in Years; Fractional if Age less than One (1) If the Age is Estimated, it is in the form xx.5With respect to the family relation variables (i.e. sibsp and parch)some relations were ignored.  The following are the definitions usedfor sibsp and parch.Sibling:  Brother, Sister, Stepbrother, or Stepsister of Passenger Aboard TitanicSpouse:   Husband or Wife of Passenger Aboard Titanic (Mistresses and Fiances Ignored)Parent:   Mother or Father of Passenger Aboard TitanicChild:    Son, Daughter, Stepson, or Stepdaughter of Passenger Aboard TitanicOther family relatives excluded from this study include cousins,nephews/nieces, aunts/uncles, and in-laws.  Some children travelledonly with a nanny, therefore parch=0 for them.  As well, sometravelled with very close friends or neighbors in a village, however,the definitions do not support such relations.<jupyter_code># This Python 3 environment comes with many helpful analytics libraries 
# installed
# It is defined by the kaggle/python docker image: 
# https://github.com/kaggle/docker-python
# For example, here's several helpful packages to load in 
from IPython.display import display
from matplotlib import pyplot as plt
from sklearn.decomposition import PCA
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_extraction import DictVectorizer
from sklearn.metrics import f1_score, accuracy_score, precision_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import (
    FunctionTransformer, OneHotEncoder, StandardScaler,
    MinMaxScaler, MaxAbsScaler, Imputer
)
from sklearn.svm import SVC

import numpy as np # linear algebra
import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)
import sklearn

# Input data files are available in the "../input/" directory.
# For example, running this (by clicking run or pressing Shift+Enter) 
# will list the files in the input directory

import sh
print(sh.ls('./input'))
Y_test =  pd.read_csv('./input/genderclassmodel.csv')
test =  pd.read_csv('./input/test.csv')
train =  pd.read_csv('./input/train.csv')

print('test set has %s rows and %s columns.' % test.shape)
print('test set has %s rows and %s columns.' % Y_test.shape)
print('train set has %s rows and %s columns.' % train.shape)
train.columns
def prepare_df(df, p_unique=0.2):
    df = df.copy()
    
    size = len(df)
    
    df = pd.get_dummies(df, drop_first=True)
    cols = list(df.columns)
    
    for c in cols:
        if df[c].dtype == object:
            if df[c].nunique()/size > p_unique:
                df.drop(c, axis=1, inplace=True)
                continue
            
    return df
pca = make_pipeline(
    Imputer(missing_values='NaN', strategy='mean', axis=0),
    StandardScaler(),
    PCA(random_state=42)
)

pca.fit_transform(prepare_df(train.drop('Survived', axis=1)), train.Survived);
pca.get_params()
train_ = train.copy()

train_.drop(['PassengerId', 'Name', 'Ticket', 'Cabin'], axis=1, inplace=True)
# train_.dropna(inplace=True)

X_train = prepare_df(train_.drop('Survived', axis=1))
y_train = train_[['Survived']]
test_ = pd.concat([Y_test, test.iloc[:, 1:]], axis=1)

test_.drop(['PassengerId', 'Name', 'Ticket', 'Cabin'], axis=1, inplace=True)
# test_.dropna(inplace=True)

X_test = prepare_df(test_.drop('Survived', axis=1))
y_test = test_[['Survived']]
test_.columns
print('X_test set has %s rows and %s columns.' % X_test.shape)
print('y_test set has %s rows and %s columns.' % y_test.shape)
print('X_train set has %s rows and %s columns.' % X_train.shape)
print('Y_train set has %s rows and %s columns.' % y_train.shape)
# , MaxAbsScaler
pipeline_rfc = make_pipeline(
    Imputer(missing_values='NaN', strategy='mean', axis=0),
    MaxAbsScaler(),
    RandomForestClassifier(random_state=42)
)

pipeline_svc = make_pipeline(
    Imputer(missing_values='NaN', strategy='mean', axis=0),
    MaxAbsScaler(),
    SVC()
)
'''
<class 'sklearn.ensemble.forest.RandomForestClassifier'>
f1 score
0.768558951965
precision score
1.0
accuracy score
0.873205741627


<class 'sklearn.svm.classes.SVC'>
f1 score
0.873015873016
precision score
0.990990990991
accuracy score
0.923444976077
'''
for title, model in {
    'RandomForestClassifier': pipeline_rfc, 
    'SVC': pipeline_svc
}.items():
    print('\n%s' % model)
    model.fit(X_train, y_train)

    predicted = model.predict(X_test)
    print('f1 score')
    print(f1_score(y_test, predicted))
    print('precision score')
    print(precision_score(y_test, predicted))
    print('accuracy score')
    print(accuracy_score(y_test, predicted))
    f_name = 'output/%s.csv' % title
    
    # np.savetxt(f_name, predicted)
    # print('%s saved!' % f_name)<jupyter_output>Pipeline(memory=None,
     steps=[('imputer', Imputer(axis=0, copy=True, missing_values='NaN', strategy='mean', verbose=0)), ('maxabsscaler', MaxAbsScaler(copy=True)), ('randomforestclassifier', RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',
            max_depth=None, max_features='auto', max_lea...stimators=10, n_jobs=1,
            oob_score=False, random_state=42, verbose=0, warm_start=False))])
f1 score
0.7491408934707905
precision score
0.7266666666666667
accuracy score
0.8253588516746412

Pipeline(memory=None,
     steps=[('imputer', Imputer(axis=0, copy=True, missing_values='NaN', strategy='mean', verbose=0)), ('maxabsscaler', MaxAbsScaler(copy=True)), ('svc', SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
  decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',
  max_iter=-1, probability=False, random_state=None, shrinking=True,
  tol=0.001, verbose=False))])
f1 score
0.962457337883959
precision score
0.9276315789473685
acc[...]
{'fraction_non_alphanumeric': 0.11680199875078076, 'fraction_numerical': 0.03357276702061212, 'mean_word_length': 4.414201183431953, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 7, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '622907', 'license': 'MIT', 'n_tokens_mistral': 2261, 'n_tokens_neox': 2106, 'n_words': 589, 'path': 'titanic/Titanic.ipynb', 'repo_name': 'xmnlab/kaggle', 'size': '6404'}