<jupyter_start><jupyter_text>ML Pipeline PreparationFollow the instructions below to help you create your ML pipeline. 1. Import libraries and load data from database.- Import Python libraries- Load dataset from database with [`read_sql_table`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_sql_table.html)- Define feature and target variables X and Y<jupyter_code># import libraries
# import libraries
import pandas as pd
import numpy as np
import pickle
import sys

import re

import nltk
nltk.download('punkt')
nltk.download('wordnet')

from nltk.tokenize import word_tokenize
from nltk.stem.wordnet import WordNetLemmatizer

from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.multioutput import MultiOutputClassifier
from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.metrics import accuracy_score, f1_score, precision_score, recall_score, classification_report, confusion_matrix


from sqlalchemy import create_engine
# load data from database
engine = create_engine('sqlite:///DisasterDatabase.db')
df = pd.read_sql_table('DisasterTableNew',engine)
X = df.message
Y = df.drop(columns=['id', 'message', 'original', 'genre'])<jupyter_output><empty_output><jupyter_text>2. Write a tokenization function to process your text data<jupyter_code>def tokenize(text):
    ''' cleanes tokens from text data by performing the below operations
        - Removes punctuation
        - Lemmatizes tokens
        - Converts to lowercase and removes extra whitespace
    Args: 
        text (str)
    Returns:
        clean_tokens (list): list of tokens
    '''
    # remove punctuation
    text = re.sub(r"[^a-zA-Z0-9]", " ", text)
    
    # tokenize
    tokens = word_tokenize(text)
    
    # initalize lemmatizer
    lemmatizer = WordNetLemmatizer()
    
    clean_tokens = []
    for token in tokens:
        clean_tokens.append(lemmatizer.lemmatize(token).lower().strip())
        
    return clean_tokens<jupyter_output><empty_output><jupyter_text>3. Build a machine learning pipelineThis machine pipeline should take in the `message` column as input and output classification results on the other 36 categories in the dataset. You may find the [MultiOutputClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.multioutput.MultiOutputClassifier.html) helpful for predicting multiple target variables.<jupyter_code>pipeline = pipeline = Pipeline([
        ('vect', CountVectorizer(tokenizer=tokenize)),
        ('tfidf', TfidfTransformer()),
        ('clf', MultiOutputClassifier(DecisionTreeClassifier()))
    ])<jupyter_output><empty_output><jupyter_text>4. Train pipeline- Split data into train and test sets- Train pipeline<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.2, random_state=42)

pipeline.fit(X_train,y_train)<jupyter_output><empty_output><jupyter_text>5. Test your modelReport the f1 score, precision and recall for each output category of the dataset. You can do this by iterating through the columns and calling sklearn's `classification_report` on each.<jupyter_code>y_pred=pipeline.predict(X_test)

category_names=Y.columns

for i in range(y_test.shape[1]):
    print('Column {}, {}'.format(i, category_names[i]))
    print(classification_report(y_test.iloc[:,i], y_pred[:,i]))<jupyter_output>Column 0, related
             precision    recall  f1-score   support

          0       0.48      0.43      0.46      1266
          1       0.82      0.84      0.83      3938
          2       0.42      0.62      0.51        40

avg / total       0.74      0.74      0.74      5244

Column 1, request
             precision    recall  f1-score   support

          0       0.91      0.92      0.91      4349
          1       0.57      0.54      0.56       895

avg / total       0.85      0.85      0.85      5244

Column 2, offer
             precision    recall  f1-score   support

          0       1.00      1.00      1.00      5218
          1       0.00      0.00      0.00        26

avg / total       0.99      0.99      0.99      5244

Column 3, aid_related
             precision    recall  f1-score   support

          0       0.74      0.74      0.74      3113
          1       0.63      0.63      0.63      2131

avg / total       0.70      0.70      0.70      5244

Column 4, med[...]<jupyter_text>6. Improve your modelUse grid search to find better parameters.<jupyter_code>parameters = {
        'vect__stop_words':['english'],
        'vect__ngram_range':[(1,2)],
        'vect__max_features':[40000],
        'clf__estimator__random_state':[1,33],
        'clf__estimator__max_depth':[5, 10],
        'clf__estimator__min_samples_split':[4, 7]
    }
    
cv = GridSearchCV(pipeline, parameters, cv=3)  

cv.fit(X_train,y_train)
cv.best_params_
prediction= cv.predict(X_test)<jupyter_output><empty_output><jupyter_text>7. Test your modelShow the accuracy, precision, and recall of the tuned model.  Since this project focuses on code quality, process, and  pipelines, there is no minimum performance metric needed to pass. However, make sure to fine tune your models for accuracy, precision and recall to make your project stand out - especially for your portfolio!<jupyter_code>print('Accuracy:', accuracy_score(y_test.values.flatten(), prediction.flatten()))
print('F1 score:', f1_score(y_test.values.flatten(), prediction.flatten(),average='weighted'))
print('Recall:', recall_score(y_test.values.flatten(), prediction.flatten(),average='weighted'))
print('Precision:', precision_score(y_test.values.flatten(), prediction.flatten(),average='weighted'))
print('\n clasification report on the folds:\n', classification_report(y_test.values.flatten(),prediction.flatten()))
print('\n confussion matrix:\n',confusion_matrix(y_test.values.flatten(), prediction.flatten()))<jupyter_output>Accuracy: 0.945705144504
F1 score: 0.942441596754
Recall: 0.945705144504
Precision: 0.941289341745

 clasification report on the folds:
              precision    recall  f1-score   support

          0       0.96      0.98      0.97    172321
          1       0.74      0.57      0.65     16423
          2       0.25      0.03      0.05        40

avg / total       0.94      0.95      0.94    188784


 confussion matrix:
 [[169093   3228      0]
 [  6980   9440      3]
 [     1     38      1]]<jupyter_text>8. Try improving your model further. Here are a few ideas:* try other machine learning algorithms* add other features besides the TF-IDF<jupyter_code>pipeline_alt = Pipeline([
        ('vect', CountVectorizer(tokenizer=tokenize)),
        ('tfidf', TfidfTransformer()),
        ('clf', MultiOutputClassifier(AdaBoostClassifier()))
    ])


parameters_alt  = {
    'vect__stop_words':['english'],
    'vect__ngram_range':[(1,2)],
    'vect__max_features':[40000],
    'clf__estimator__learning_rate':[0.75, 1.0],
    'clf__estimator__n_estimators':[50, 75]
}

cv_alt  = GridSearchCV(pipeline_alt , parameters_alt, cv=3) 

cv_alt.fit(X_train,y_train) 
prediction_alt= cv_alt.predict(X_test)
print('Accuracy:', accuracy_score(y_test.values.flatten(), prediction_alt.flatten()))
print('F1 score:', f1_score(y_test.values.flatten(), prediction_alt.flatten(),average='weighted'))
print('Recall:', recall_score(y_test.values.flatten(), prediction_alt.flatten(),average='weighted'))
print('Precision:', precision_score(y_test.values.flatten(), prediction_alt.flatten(),average='weighted'))
print('\n clasification report on the folds:\n', classification_report(y_test.values.flatten(),prediction_alt.flatten()))
print('\n confussion matrix:\n',confusion_matrix(y_test.values.flatten(), prediction_alt.flatten()))<jupyter_output>Accuracy: 0.94721480634
F1 score: 0.943687481184
Recall: 0.94721480634
Precision: 0.94278985095

 clasification report on the folds:
              precision    recall  f1-score   support

          0       0.96      0.98      0.97    172321
          1       0.76      0.57      0.65     16423
          2       0.20      0.05      0.08        40

avg / total       0.94      0.95      0.94    188784


 confussion matrix:
 [[169427   2894      0]
 [  7025   9390      8]
 [     0     38      2]]<jupyter_text>9. Export your model as a pickle file<jupyter_code>model_filepath = 'model.pkl'

try:
    f = open(model_filepath, 'wb')
    pickle.dump(cv_alt, f)
    print('model saved at {}'.format(model_filepath))
except IOError:
    print(IOError.message)<jupyter_output>model saved at model.pkl
{'fraction_non_alphanumeric': 0.10216861881015887, 'fraction_numerical': 0.05891221152731068, 'mean_word_length': 3.196593673965937, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 17, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '340376', 'license': 'MIT', 'n_tokens_mistral': 3017, 'n_tokens_neox': 2748, 'n_words': 679, 'path': 'pipeline/ML Pipeline Preparation.ipynb', 'repo_name': 'ian0549/Disaster-Response-App', 'size': '8623'}