<jupyter_start><jupyter_text>IntroductionIn this tutorial, we will go through a typical ML workflow with Foreshadow using the [Titanic](https://www.kaggle.com/c/titanic) dataset from Kaggle. Getting StartedTo get started with foreshadow, install the package using `pip install foreshadow`. This will also install the dependencies. Now create a simple python script that uses all the defaults with Foreshadow. Note that Foreshadow requires `Python >=3.6, <4.0`. First import foreshadow related classes. Also import sklearn, pandas and numpy packages.<jupyter_code># Import required libraries
import re
import pandas as pd 
import numpy as np

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
from sklearn.metrics import roc_auc_score
from sklearn.metrics import accuracy_score

from foreshadow import Foreshadow
from foreshadow.intents import IntentType
from foreshadow.utils import ProblemType
from foreshadow.estimators import AutoEstimator
from foreshadow.concrete.internals.cleaners.customizable_base import CustomizableBaseCleaner

# Set the random state
RANDOM_SEED = 10001
np.random.seed(RANDOM_SEED)<jupyter_output><empty_output><jupyter_text>Load the dataset<jupyter_code># Load the data
titanic = pd.read_csv('data/titanic_train.csv')
print('Data Shape: ' + str(titanic.shape))
X_train, X_test, y_train, y_test = train_test_split(titanic.drop('Survived', axis=1), titanic['Survived'],
                                                    train_size=0.8, test_size=0.2)
titanic.head(5)<jupyter_output>Data Shape: (891, 12)<jupyter_text>Model Training Iteration 1 - Use a simple LogisticRegression Model<jupyter_code># Define a function to measure the performance of the model to be trained


def measure(model, X_test, y_test):
    y_pred = model.predict(X_test)
    print('Accuracy = %5.2f' % accuracy_score(y_test, y_pred))
    print('Classification Report:\n')
    print(classification_report(y_test, y_pred))
shadow1 = Foreshadow(problem_type=ProblemType.CLASSIFICATION, estimator=LogisticRegression(random_state=RANDOM_SEED))
shadow1.fit(X_train, y_train)
measure(shadow1, X_test, y_test)<jupyter_output>2020-03-26 17:46:39,810 - foreshadow - INFO - 26003 - Starting cleaning rows...
2020-03-26 17:46:39,813 - foreshadow - INFO - 26003 - Ending cleaning rows...
2020-03-26 17:46:40,133 - foreshadow - INFO - 26003 - Exported processed data to processed_test_data.csv<jupyter_text>You might be curious how Foreshadow handled the input data. Let's take a look<jupyter_code>shadow1.get_data_summary()<jupyter_output><empty_output><jupyter_text>Foreshadow use a machine learning model to identify the 'intent' of features. 3 intents are supported as of v1.0 and they are 'Categorical', 'Numeric' and 'Text'. Foreshadow will transform the features intelligently according to its intent and statistics. Features not belonging to these three are tagged as 'Droppable'. For example, the PassengerId is droppable since it has a unique value for each row and will not provide any signal to the model. Also in the above table, 'Label' in the intent row indicate that is the target column.  Model Training Iteration 2 - Override   From the table above, we found that the 'Ticket' column is tagged with Numeric intent, but the column seems represent the ticket identification number. In order to avoid confusing the model, we can mark the column with Droppable intent. We similarly apply this to the Cabin and Name column.<jupyter_code>shadow1 = Foreshadow(problem_type=ProblemType.CLASSIFICATION, estimator=LogisticRegression(random_state=RANDOM_SEED))
shadow1.override_intent('Ticket', IntentType.DROPPABLE)
shadow1.override_intent('Cabin', IntentType.DROPPABLE)
shadow1.override_intent('Name', IntentType.DROPPABLE)
shadow1.fit(X_train, y_train)
measure(shadow1, X_test, y_test)<jupyter_output>2020-03-26 17:46:44,227 - foreshadow - INFO - 26003 - Starting cleaning rows...
2020-03-26 17:46:44,230 - foreshadow - INFO - 26003 - Ending cleaning rows...
2020-03-26 17:46:44,324 - foreshadow - INFO - 26003 - Exported processed data to processed_test_data.csv<jupyter_text>Model Training Iteration 3 - AutoEstimator   Instead of trying one estimator, we can leverage AutoEstimator to search ML models and hyper-parameters. When we do not provide an estimator, Foreshadow will create the AutoEstimator automatically.<jupyter_code>shadow2 = Foreshadow(problem_type=ProblemType.CLASSIFICATION, allowed_seconds=300, random_state=RANDOM_SEED, n_jobs=-1)
shadow2.override_intent('Ticket', IntentType.DROPPABLE)
shadow2.override_intent('Cabin', IntentType.DROPPABLE)
shadow2.override_intent('Name', IntentType.DROPPABLE)
shadow2.fit(X_train, y_train)
measure(shadow2, X_test, y_test)<jupyter_output>2020-03-26 17:46:49,523 - foreshadow - INFO - 26003 - Exported processed data to processed_training_data.csv<jupyter_text>Model Training Iteration 4 - Data Cleaner   A closer look at the Name column found that people's titles are embedded. Since the title represent people's social status, that might have an impact to the model performance. Let's try to extract that information.<jupyter_code>class TitleExtractor(CustomizableBaseCleaner):
    def __init__(self):
        super().__init__(
            transformation=lambda row: row if row is None else re.search(' ([A-Za-z]+)\.', str(row)).group(0)
        )

    def metric_score(self, X: pd.DataFrame) -> float:
        return 1 if X.columns[0] == "Name" else 0

shadow3 = Foreshadow(problem_type=ProblemType.CLASSIFICATION, allowed_seconds=300, random_state=RANDOM_SEED, n_jobs=-1)
shadow3.override_intent('Ticket', IntentType.DROPPABLE)
shadow3.override_intent('Cabin', IntentType.DROPPABLE)
shadow3.register_customized_data_cleaner(data_cleaners=[TitleExtractor])
shadow3.fit(X_train, y_train)
measure(shadow3, X_test, y_test)
shadow3.get_data_summary()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08404918308910224, 'fraction_numerical': 0.03671888159002863, 'mean_word_length': 5.717194570135747, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 27, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '553840', 'license': 'Apache-2.0', 'n_tokens_mistral': 1972, 'n_tokens_neox': 1814, 'n_words': 552, 'path': 'examples/titanic/titanic.ipynb', 'repo_name': 'adithyabsk/foreshadow', 'size': '5937'}