<jupyter_start><jupyter_text>Note: this notebook is meant to be a demo of some of the capabilities of **`oo-learning`** (https://github.com/shane-kercheval/oo-learning); it is not meant to show the best approach to exploring/cleaning/modeling this particular dataset. Also, with most graphs (e.g. correlations/box-plots/etc.) I will spend very little time commenting on the significance of any interesting or patterns. Again, the intent is to show a demo, not a guide to data analysis.   Set Up Environment<jupyter_code># !pip install --upgrade pip
# !pip install oolearning --upgrade
# !pip install hyperopt
# !pip install bayesian-optimization
# !pip install lightgbm
from oolearning import *
import pandas as pd
import matplotlib.pyplot as plt

%matplotlib inline
pd.set_option('display.max_colwidth', -1)
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)
width = 9
plt.rcParams['figure.figsize'] = [width, width/1.333]<jupyter_output><empty_output><jupyter_text>Import Data **`ExploreClassificationDataset`** is a class that gives convenience while exploring a new (classification) dataset by providing common functionality frequently needed during standard exploration. Below, the class is initialized from a csv file (but you can also initialize from an existing pandas DataFarme using the constructor, **`ExploreClassificationDataset(dataset, target_variable)`**.Additionally, sometimes we have a numeric target (even though the target is logically categorical), that we would like to change in order to make the outcome more specific. In this example, we load in the **`titanic`** dataset (https://www.kaggle.com/c/titanic/data), and change the target variable (**`Survived`**) from **`1`'s** & **`0`'s**, to **`lived`**/**`died`**.<jupyter_code>csv_file = '../data/titanic.csv'
target_variable = 'Survived'
target_mapping = {0: 'died', 1: 'lived'}  # so we can convert from numeric to categoric

explore = ExploreClassificationDataset.from_csv(csv_file_path=csv_file,
                                                target_variable=target_variable,
                                                map_numeric_target=target_mapping)<jupyter_output><empty_output><jupyter_text>Getting a General Understanding of the Data<jupyter_code>explore.dataset.head()  # we can access the data directly by `.dataset`<jupyter_output><empty_output><jupyter_text>The first thing that I noticed is **`Pclass`** is imported as numeric feature, but it might be better suited as categoric, so let's convert it via **`.set_as_categoric()`**. Note that **`Pclass`** will now show up under categoric values when we do **`.categoric_summary()`**, below.<jupyter_code># Description of data says (https://www.kaggle.com/c/titanic/data):
# pclass: A proxy for socio-economic status (SES)
# 1st = Upper
# 2nd = Middle
# 3rd = Lower

# let's convert Pclass to categoric variable
explore.set_as_categoric(feature='Pclass', mapping={1: 'Upper', 2: 'Middle', 3: 'Lower'})<jupyter_output><empty_output><jupyter_text>Let's explore the numeric columns.<jupyter_code>explore.numeric_summary()<jupyter_output><empty_output><jupyter_text>A couple of things we might note, for example:    A) `Age` has missing values.    B) `Fare` has `15` zeros, which we might assume is equivalent to a missing/null value. (Tickets probably weren't free.)    C) `PassengerId` will not be helpful. Now let's explore the categoric columns.<jupyter_code>explore.categoric_summary()<jupyter_output><empty_output><jupyter_text>For these columns, I might note, for example, that `Name`, `Ticket`, and `Cabin` all have a very high number of unique values (`Cabin` also have a high number of null values). This obviously make sense, but also means we might not want to use these columns._(Note: in https://ahmedbesbes.com/how-to-score-08134-in-titanic-kaggle-challenge.html, the author describes how he cleverly extracts the title associated with the name, where applicable. That is a great idea, but outside the scope of this demo.)_<jupyter_code>explore.drop(columns=['PassengerId', 'Name', 'Ticket', 'Cabin'])  # drop/remove specific columns
explore.dataset.head()<jupyter_output><empty_output><jupyter_text>Also, the `__str__` function is implemented in this class so we can simply print it out to get a quick view of the numeric and categoric summaries.<jupyter_code>print(explore)<jupyter_output>Target Variable: Survived
================

Numeric Summary
===============
       count  nulls  perc_nulls  num_zeros  perc_zeros    mean  st_dev  coef of var  skewness  kurtosis   min    10%     25%     50%   75%     90%      max
Age    714    177    0.199       0          0.000       29.699  14.526  0.489        0.389     0.178     0.42  14.00  20.125  28.000  38.0  50.000  80.000 
SibSp  891    0      0.000       608        0.682       0.523   1.103   2.108        3.695     17.880    0.00  0.00   0.000   0.000   1.0   1.000   8.000  
Parch  891    0      0.000       678        0.761       0.382   0.806   2.112        2.749     9.778     0.00  0.00   0.000   0.000   0.0   2.000   6.000  
Fare   891    0      0.000       15         0.017       32.204  49.693  1.543        4.787     33.398    0.00  7.55   7.910   14.454  31.0  77.958  512.329


Categoric Summary
=================
          count  nulls  perc_nulls    top  unique  perc_unique
Survived  891    0      0.000       died   [...]<jupyter_text>Diving In  Numeric FeaturesLet's dive in a little more into the characteristics of our numeric features.  CorrelationsA good place to start is with an overall idea of how correlated our numeric features are.<jupyter_code>explore.plot_correlation_heatmap()
explore.plot_scatterplot_numerics()<jupyter_output><empty_output><jupyter_text>FeaturesNow let's take a look at each numeric feature, by itself, as well as how it relates to our target variable.  Age<jupyter_code>explore.plot_boxplot(numeric_feature='Age')
explore.plot_against_target(feature='Age')
explore.plot_histogram(numeric_feature='Age')
explore.plot_histogram_against_target(numeric_feature='Age')<jupyter_output><empty_output><jupyter_text>SibSp<jupyter_code>explore.plot_histogram(numeric_feature='SibSp')<jupyter_output><empty_output><jupyter_text>Parch<jupyter_code>explore.plot_histogram(numeric_feature='Parch')
explore.plot_against_target(feature='Parch')<jupyter_output><empty_output><jupyter_text>Above, we saw there was moderate correlation of **`Parch`** and  **`SibSp`**, let's look at a scatter plot to see if it shows anything useful.<jupyter_code>explore.plot_scatterplot_numerics(numeric_columns=['Parch', 'SibSp'])<jupyter_output><empty_output><jupyter_text>Fare<jupyter_code>explore.plot_against_target(feature='Fare')
explore.plot_histogram(numeric_feature='Fare')
explore.plot_histogram_against_target(numeric_feature='Fare')<jupyter_output><empty_output><jupyter_text>Categoric FeaturesLet's dive in a little more into the characteristics of our categoric features.  Pclass We can see the unique values (and correspond frequencies) for a given feature via **`.unique_values()`**:<jupyter_code>explore.unique_values(categoric_feature='Pclass')<jupyter_output><empty_output><jupyter_text>Or we can visualize it via **`.plot_unique_values()`**:<jupyter_code>explore.plot_unique_values(categoric_feature='Pclass')<jupyter_output><empty_output><jupyter_text>Often, a categoric feature has inherent order. With `Pclass`, for example, we can think of the order as: `Lower < Middle < Upper`.In this case, it would be convenient to see the data in that order.Using the **`sort_by_feature`** parameter in the `.unique_values()` method, we can order by feature.<jupyter_code>explore.unique_values(categoric_feature='Pclass', sort_by_feature=True)<jupyter_output><empty_output><jupyter_text>But this didn't do **exactly** what we want. It is in order, but is sorting in (reverse) alphabetical order. It's not sorting by what we think of as the logical order of `Pclass`, because we haven't specified any order.But we can do this for categoric fatures via **`.set_level_order()`**:<jupyter_code># now set the order of categorical
explore.set_level_order(categoric_feature='Pclass',
                        levels=['Lower', 'Middle', 'Upper'])
explore.unique_values(categoric_feature='Pclass', sort_by_feature=True)
explore.plot_unique_values(categoric_feature='Pclass', sort_by_feature=True)
explore.plot_against_target(feature='Pclass')<jupyter_output><empty_output><jupyter_text>Sex<jupyter_code>explore.plot_unique_values(categoric_feature='Sex')
explore.plot_against_target(feature='Sex')<jupyter_output><empty_output><jupyter_text>Embarked<jupyter_code>explore.plot_unique_values(categoric_feature='Embarked')
explore.plot_against_target(feature='Embarked')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10609171800136892, 'fraction_numerical': 0.033424595026237736, 'mean_word_length': 4.923648648648649, 'pattern_counts': {'":': 0, '<': 82, '<?xml version=': 0, '>': 80, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '373898', 'license': 'MIT', 'n_tokens_mistral': 3000, 'n_tokens_neox': 2752, 'n_words': 804, 'path': 'examples/classification-titanic/1-Exploring the Titanic Dataset.ipynb', 'repo_name': 'shane-kercheval/oo-learning', 'size': '8766'}