<jupyter_start><jupyter_text>Model: Decision Tree Model Following Reference With and Without Rebalancing Decision Trees on the Titanic Data Set http://benalexkeen.com/decision-tree-classifier-in-python-using-scikit-learn/* features = ['pclass', 'sex', 'age', 'sibsp', 'parch', 'fare']* Accuracy Score = 0.83240223463687146 The above Kaggle Kernel shows how Decision Trees may be applied to the Titanic data. The author shows an accuracy of 0.8324.  The author does not specify the random_state so it is impossible to reproduce his work exactly. What is interesting is that the accuracy calculated and cross validation scores are significantly different. I don't understand why this is the case.   Initialization<jupyter_code>%run init.ipynb
from sklearn.impute import SimpleImputer
from sklearn.tree import (DecisionTreeClassifier, 
                          export_graphviz)

from sklearn.preprocessing import scale

import great_expectations as ge

RANDOM_STATE = 1

from progressbar import ProgressBar
import seaborn as sns<jupyter_output><empty_output><jupyter_text>For more information on progressbar. Look here https://pypi.org/project/progressbar2/  Extract Clean Data **Separate data into X (features) and y (label) from the raw data**This is exactly what Ben did in his notebook.<jupyter_code>Xy = pd.read_csv('../data/raw/train.csv')
Xy.columns = Xy.columns.str.lower()

features = ['passengerid', 'pclass', 'sex', 'age', 'sibsp', 'parch', 'fare', 'survived']

Xy = Xy[features].dropna().set_index('passengerid')
Xy.sex = Xy.sex.map({'male':0, 'female':1})         # Convert sex to an integer via mapping. 

X = Xy.drop('survived', axis=1)
y = Xy['survived']<jupyter_output><empty_output><jupyter_text>**List of Features**<jupyter_code>X<jupyter_output><empty_output><jupyter_text>Train Test Split Data  Split data into train and test with Near Miss Balancing.  https://imbalanced-learn.readthedocs.io/en/stable/auto_examples/under-sampling/plot_illustration_nearmiss.htmlnearmiss-1<jupyter_code>from imblearn.under_sampling import NearMiss

nm = NearMiss(version=3, random_state = RANDOM_STATE)
X_resampled_nm1, y_resampled_nm1 = nm.fit_resample(X.reset_index(), y)

df_X_resampled = pd.DataFrame(X_resampled_nm1, columns=X.reset_index().columns).set_index('passengerid')
df_y_resampled = pd.DataFrame(y_resampled_nm1, index=df_X_resampled.index, columns=['survived'])

df_X_resampled = df_X_resampled.sort_index()
df_y_resampled = df_y_resampled.sort_index()
df_X_resampled
df_y_resampled<jupyter_output><empty_output><jupyter_text>Definitions<jupyter_code>def check_accuracy_by_random_sampling(X,y):
    
    accuracy = []
    n_random=1000
    RANDOM_STATE_RANGE = list(range(0,n_random,1))
    len(RANDOM_STATE_RANGE)
    verbose = False

    pbar = ProgressBar(maxval=n_random).start()

    for i, random_state in enumerate(RANDOM_STATE_RANGE, start=0):
        
        X_train, X_test, y_train, y_test = train_test_split(X, 
                                                            y, 
                                                            test_size=0.2, 
                                                            random_state=random_state)

        model = DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,
                        max_features=None, max_leaf_nodes=None,
                        min_impurity_split=1e-07, min_samples_leaf=1,
                        min_samples_split=2, min_weight_fraction_leaf=0.0,
                        presort=False, random_state=random_state, splitter='best');

        _ = model.fit(X_train, y_train);

        ii_y_pred, ii_predicted_accuracy_score, ii_cv_scores = pm.calc_model_rst_table_metrics(model, X_train, y_train, X_test, y_test, 
                                                      model_name='dtree_reference', cv=5, verbose=verbose)
    
   
        accuracy.append( [random_state,
                              ii_predicted_accuracy_score, 
                              np.mean(ii_cv_scores['test_accuracy']),
                              np.std(ii_cv_scores['test_accuracy'])])

        pbar.update(i)

    pbar.finish()
    
    return np.array(accuracy)<jupyter_output><empty_output><jupyter_text>Decision Tree Classifier without rebalancing<jupyter_code>accuracy = check_accuracy_by_random_sampling(X,y)
df_accuracy =  pd.DataFrame(accuracy[:,1:3], columns=['Test', 'Cross Val'])
df_accuracy.describe()
fig, ax = plt.subplots(figsize=(12, 12));

labels = ['test', 'cross_val']
sns.set(font_scale = 2)  # This changes the font size off all fonts by the same scale. 

for ii, a in enumerate(np.transpose(accuracy[:,1:3])):
    _ = sns.distplot(a,  ax=ax,  kde=True, hist=False, kde_kws={"shade": True});
   
fig.legend(labels=labels, fontsize=20, loc='upper left');<jupyter_output><empty_output><jupyter_text>Decision Tree Classifier with rebalancing<jupyter_code>accuracy_rebalanced = check_accuracy_by_random_sampling(df_X_resampled,df_y_resampled)
df_accuracy_rebalanced =  pd.DataFrame(accuracy_rebalanced[:,1:3], columns=['Test', 'Cross Val'])
df_accuracy_rebalanced.describe()
fig, ax = plt.subplots(figsize=(12, 12));

labels = ['test', 'cross_val']
sns.set(font_scale = 2)  # This changes the font size off all fonts by the same scale. 

for ii, a in enumerate(np.transpose(accuracy_rebalanced[:,1:3])):
    _ = sns.distplot(a,  ax=ax,  kde=True, hist=False, kde_kws={"shade": True});
   
fig.legend(labels=labels, fontsize=20, loc='upper left');<jupyter_output><empty_output><jupyter_text>Visualize the Decision Tree<jupyter_code>feature_names=X.columns

tree_name = 'tree_ben_keen'
export_graphviz(model, out_file=f'{tree_name}.dot', feature_names=feature_names)

from subprocess import call
call(['dot', '-T', 'png', f'{tree_name}.dot', '-o', f'{tree_name}.png'])<jupyter_output><empty_output><jupyter_text>Prepare Submission<jupyter_code>X = pd.read_csv('../data/processed/holdout.csv', index_col='passengerid')
X_test_kaggle_public = transform_X(X).reindex(X_test.columns, axis=1).fillna(0)
X_test_kaggle_public
y_pred = (pd.Series(model.predict(X_test_kaggle_public), 
                   index=X.index, name='Survived').to_frame().sort_index()
         )

y_pred.index.names = ['PassengerId']

y_submission = (pd.read_csv('../data/raw/gender_submission.csv')
                .set_index('PassengerId')
               )
y_submission
(y_pred.index == y_submission.index).all()
y_pred.index.names == y_submission.index.names
(y_pred.columns == y_submission.columns).all()
filename = 'dtree_model_2.csv'
y_pred.to_csv(filename)

y_pred_file = (pd.read_csv(filename)
                .set_index('PassengerId')
               )

(y_pred_file.index == y_submission.index).all()
y_pred_file.index.names == y_submission.index.names
(y_pred_file.columns == y_submission.columns).all()<jupyter_output><empty_output><jupyter_text>Simplify Model<jupyter_code>from yellowbrick.features import Rank1D
# Instantiate the 1D visualizer with the Sharpiro ranking algorithm
visualizer = Rank1D(algorithm='shapiro', size=(1000,1000))

visualizer.fit(X, y, )           # Fit the data to the visualizer
visualizer.transform(X)        # Transform the data
visualizer.show();          # Finalize and render the figure<jupyter_output><empty_output><jupyter_text>https://www.districtdatalabs.com/visualize-data-science-pipeline-with-yellowbrick<jupyter_code>from cycler import cycler
import matplotlib as mpl
from yellowbrick.model_selection import FeatureImportances
# Create a new figure
mpl.rcParams['axes.prop_cycle'] = cycler('color', ['red'])
fig = plt.gcf()
fig.set_size_inches(20,20)
ax = plt.subplot(311)
labels = X.columns

viz = FeatureImportances(model, ax=ax, labels=labels, relative=False, absolute=True)
ax.spines['right'].set_visible(False)
ax.spines['top'].set_visible(False)
ax.grid(False)

# Fit and display
viz.fit(X, y)
viz.poof()
from sklearn.metrics import confusion_matrix

pd.DataFrame(
    confusion_matrix(y_test, y_pred),
    columns=['Predicted Not Survival', 'Predicted Survival'],
    index=['True Not Survival', 'True Survival']
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11739990113692536, 'fraction_numerical': 0.011616411270390509, 'mean_word_length': 4.251784555483452, 'pattern_counts': {'":': 2, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '544460', 'license': 'MIT', 'n_tokens_mistral': 2856, 'n_tokens_neox': 2728, 'n_words': 515, 'path': 'notebooks/model__dtree_reference__2019-11-05.ipynb', 'repo_name': 'bkraft4257/kaggle_titanic', 'size': '8092'}