<jupyter_start><jupyter_text>ML Pipeline PreparationFollow the instructions below to help you create your ML pipeline. 1. Import libraries and load data from database.- Import Python libraries- Load dataset from database with [`read_sql_table`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_sql_table.html)- Define feature and target variables X and Y<jupyter_code>!pip install nltk
# import libraries
import numpy as np
import pandas as pd
import os
from sqlalchemy import create_engine

import nltk
nltk.download('punkt')
nltk.download('stopwords')

from sklearn.multioutput import MultiOutputClassifier
from sklearn.linear_model import *
from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_extraction.text import *
from sklearn.metrics import *
from sklearn.model_selection import GridSearchCV

print(os.listdir())
import matplotlib.pyplot as plt

%matplotlib inline
# load data from database
engine = create_engine('sqlite:///P1Data.db')
df = pd.read_sql_table('P1Data', engine)
X = 'message'
Y = [col for col in df.columns.tolist() if col not in ['id', 'message', 'original', 'genre']]
print("X: ", X)
print("T: ", Y)
# Survey target variables:

for col in Y:
    if df[col].nunique() != 2:
        print(col, ' : ', df[col].nunique())
# Process the target variable:
df.drop(['child_alone'], axis=1, inplace=True)

index = df[df['related']==2].index
print(len(index))
df.drop(index, inplace=True)

Y.remove('child_alone')
df[Y].hist()<jupyter_output><empty_output><jupyter_text>2. Write a tokenization function to process your text data<jupyter_code>def tokenize(text):
    import re
    
    processed_text = re.sub(r"[^A-Za-z0-9]", " ", text.lower()) # remove special characters.
    tokens = nltk.tokenize.word_tokenize(processed_text) # into word tokens
    
    stemmer = nltk.stem.PorterStemmer()
    stemmed_words = [stemmer.stem(_word) for _word in tokens if _word not in nltk.corpus.stopwords.words("english")]
    
    return stemmed_words
# Unit test:
tokenize(df.iloc[0]['message'])<jupyter_output><empty_output><jupyter_text>3. Build a machine learning pipelineThis machine pipeline should take in the `message` column as input and output classification results on the other 36 categories in the dataset. You may find the [MultiOutputClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.multioutput.MultiOutputClassifier.html) helpful for predicting multiple target variables.<jupyter_code>df.sample(3)
pipeline = Pipeline([
    ("tfidf_vectorizer", TfidfVectorizer(tokenizer=tokenize)) , 
    ("clf", MultiOutputClassifier(RidgeClassifier(random_state=2020)))
])<jupyter_output><empty_output><jupyter_text>4. Train pipeline- Split data into train and test sets- Train pipeline<jupyter_code>from sklearn.model_selection import train_test_split

X, y = df[X], df[Y]
tra_x, val_x, tra_y, val_y = train_test_split(X, y, random_state=2020, test_size=0.3)

pipeline.fit(tra_x, tra_y)<jupyter_output><empty_output><jupyter_text>5. Test your modelReport the f1 score, precision and recall for each output category of the dataset. You can do this by iterating through the columns and calling sklearn's `classification_report` on each.<jupyter_code>val_y.head()
preds = pipeline.predict(val_x)
preds
from collections import deque

def eval_model(pipeline, val_x, val_y):
    
    accuracy, precision, recall, f1, index = deque(), deque(), deque(), deque(), deque() # placeholder for metrics
    
    preds = pipeline.predict(val_x)
    
    for i, col in enumerate(val_y.columns.tolist()):
        accuracy.append(accuracy_score(val_y[col], preds[:, i]))
        precision.append(precision_score(val_y[col], preds[:, i]))
        recall.append(recall_score(val_y[col], preds[:, i]))
        f1.append(f1_score(val_y[col], preds[:, i]))
        index.append(col)
        
    return pd.DataFrame({
        'accuracy' : accuracy ,
        'precision' : precision , 
        'recall' : recall , 
        'f1' : f1} , 
        index = index
    )
eval_metrics = eval_model(pipeline, val_x, val_y)
eval_metrics
print("Overall performance:\n{}".format(eval_metrics.mean()))<jupyter_output>Overall performance:
accuracy     0.948470
precision    0.597253
recall       0.295337
f1           0.366034
dtype: float64<jupyter_text>6. Improve your modelUse grid search to find better parameters.<jupyter_code>parameters = {
    'clf__estimator__alpha' : [0.001, 0.003, 0.01, 0.03, 0.1]
}


def f1_scorer(truth, prediction):
    assert truth.shape[1] == prediction.shape[1]
    
    f1_scores = [] 
    for i in range(truth.shape[1]):
        f1_scores.append(f1_score(truth[:, i], prediction[:, i]))
    
    return np.mean(f1_scores)

scorer = make_scorer(f1_scorer)

# Will use f1 as the target metric for CV. Due to the sparsity, accuracy is not a desirable metric
# for this task, and f1 score can balance the precision and recall.
cv = GridSearchCV(pipeline, param_grid=parameters, scoring=scorer, verbose=1, n_jobs=-1)
# Will use the whole dataset for Grid search CV.
cv_result = cv.fit(X.values, y.values)
cv_result.best_estimator_
cv_result.best_params_<jupyter_output><empty_output><jupyter_text>7. Test your modelShow the accuracy, precision, and recall of the tuned model.  Since this project focuses on code quality, process, and  pipelines, there is no minimum performance metric needed to pass. However, make sure to fine tune your models for accuracy, precision and recall to make your project stand out - especially for your portfolio!<jupyter_code>eval_metrics_using_best_params = eval_model(cv_result.best_estimator_, val_x, val_y)
eval_metrics
print("Overall performance:\n{}".format(eval_metrics_using_best_params.mean()))<jupyter_output>Overall performance:
accuracy     0.985680
precision    0.970405
recall       0.791239
f1           0.866409
dtype: float64<jupyter_text>8. Try improving your model further. Here are a few ideas:* try other machine learning algorithms* add other features besides the TF-IDF<jupyter_code>print("After the hyper-parameter tuning, the existing model performs pretty decent. It takes too much time for this grid search operation.")<jupyter_output>After the hyper-parameter tuning, the existing model performs pretty decent. It takes too much time for this grid search operation.<jupyter_text>9. Export your model as a pickle file<jupyter_code>from sklearn.externals import joblib

joblib.dump(cv_result.best_estimator_, 'best_pipeline.pkl')<jupyter_output>/opt/conda/lib/python3.7/site-packages/sklearn/externals/joblib/__init__.py:15: FutureWarning: sklearn.externals.joblib is deprecated in 0.21 and will be removed in 0.23. Please import this functionality directly from joblib, which can be installed with: pip install joblib. If this warning is raised when loading pickled models, you may need to re-serialize those models with scikit-learn 0.21+.
  warnings.warn(msg, category=FutureWarning)
{'fraction_non_alphanumeric': 0.10239055299539171, 'fraction_numerical': 0.020161290322580645, 'mean_word_length': 4.966494845360825, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '192020', 'license': 'MIT', 'n_tokens_mistral': 2291, 'n_tokens_neox': 2200, 'n_words': 641, 'path': 'notebooks/ML Pipeline Preparation.ipynb', 'repo_name': 'myx9090/UdacityDataScienceNanodegreeProject1', 'size': '6944'}