<jupyter_start><jupyter_text>ML Pipeline PreparationFollow the instructions below to help you create your ML pipeline. 1. Import libraries and load data from database.- Import Python libraries- Load dataset from database with [`read_sql_table`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_sql_table.html)- Define feature and target variables X and Y<jupyter_code>import nltk
  nltk.download(['punkt', 'wordnet','averaged_perceptron_tagger'])
# import libraries
import re
import pandas as pd
import sqlalchemy as db
from nltk.tokenize import word_tokenize
from nltk.stem import WordNetLemmatizer
import pickle

from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import GridSearchCV
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.multioutput import MultiOutputClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.metrics import classification_report
from sklearn import model_selection
from sklearn.neighbors import KNeighborsClassifier
# load data from database
engine = db.create_engine('sqlite:///InsertDatabaseName.db')
df = pd.read_sql_table('CategorisedMessages', engine)
X = df['message']
Y = df[['related', 'request', 'offer',
       'aid_related', 'medical_help', 'medical_products', 'search_and_rescue',
       'security', 'military', 'child_alone', 'water', 'food', 'shelter',
       'clothing', 'money', 'missing_people', 'refugees', 'death', 'other_aid',
       'infrastructure_related', 'transport', 'buildings', 'electricity',
       'tools', 'hospitals', 'shops', 'aid_centers', 'other_infrastructure',
       'weather_related', 'floods', 'storm', 'fire', 'earthquake', 'cold',
       'other_weather', 'direct_report']]<jupyter_output><empty_output><jupyter_text>2. Write a tokenization function to process your text data<jupyter_code>def tokenize(text):
    # Replace none-character with space
    text = re.sub('[^A-Za-z0-9]',' ',text)
    
    # Tokenize the input
    tokens = word_tokenize(text)
    
    # Initialize lemmatizer for standardize form of words
    lemmatizer = WordNetLemmatizer()
    
    # We will iterate each token in the list, lemmatize and return result
    clean_tokens = []
    for tok in tokens:
        clean_tok = lemmatizer.lemmatize(tok).lower().strip()
        clean_tokens.append(clean_tok)

    return clean_tokens<jupyter_output><empty_output><jupyter_text>3. Build a machine learning pipelineThis machine pipeline should take in the `message` column as input and output classification results on the other 36 categories in the dataset. You may find the [MultiOutputClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.multioutput.MultiOutputClassifier.html) helpful for predicting multiple target variables.<jupyter_code>pipeline = Pipeline([
    ('vect', CountVectorizer(tokenizer=tokenize)),
    ('tfidf', TfidfTransformer()),
    ('clf', MultiOutputClassifier(RandomForestClassifier()))
])<jupyter_output><empty_output><jupyter_text>4. Train pipeline- Split data into train and test sets- Train pipeline<jupyter_code>X_train, X_test, y_train, y_test = train_test_split(X, Y)
pipeline.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>5. Test your modelReport the f1 score, precision and recall for each output category of the dataset. You can do this by iterating through the columns and calling sklearn's `classification_report` on each.<jupyter_code>y_pred = pipeline.predict(X_test)
y_test.columns
df_pred = pd.DataFrame(y_pred)
df_pred.columns=y_test.columns

for column in y_test:
    print(column)
    print(classification_report(y_test[column],df_pred[column]))
pipeline.get_params()<jupyter_output><empty_output><jupyter_text>6. Improve your modelUse grid search to find better parameters.<jupyter_code>parameters = {
     'clf__estimator__n_estimators': [10,20]
}

cv = GridSearchCV(pipeline, param_grid=parameters,n_jobs=4)<jupyter_output><empty_output><jupyter_text>7. Test your modelShow the accuracy, precision, and recall of the tuned model.  Since this project focuses on code quality, process, and  pipelines, there is no minimum performance metric needed to pass. However, make sure to fine tune your models for accuracy, precision and recall to make your project stand out - especially for your portfolio!<jupyter_code>cv.fit(X_train, y_train)
y_pred = cv.predict(X_test)
df_pred2 = pd.DataFrame(y_pred)
df_pred2.columns=y_test.columns

for column in y_test:
    print(column)
    print(classification_report(y_test[column],df_pred2[column]))<jupyter_output>related
             precision    recall  f1-score   support

          0       0.64      0.38      0.47      1487
          1       0.83      0.93      0.88      5020
          2       0.50      0.45      0.47        47

avg / total       0.79      0.80      0.78      6554

request
             precision    recall  f1-score   support

          0       0.88      0.98      0.93      5432
          1       0.83      0.36      0.50      1122

avg / total       0.87      0.88      0.86      6554

offer
             precision    recall  f1-score   support

          0       1.00      1.00      1.00      6528
          1       0.00      0.00      0.00        26

avg / total       0.99      1.00      0.99      6554

aid_related
             precision    recall  f1-score   support

          0       0.71      0.88      0.78      3744
          1       0.76      0.52      0.61      2810

avg / total       0.73      0.72      0.71      6554

medical_help
             precision    recall  f1-sco[...]<jupyter_text>8. Try improving your model further. Here are a few ideas:* try other machine learning algorithms* add other features besides the TF-IDF<jupyter_code># Function that count number of adv and adj in the sentence and return back as new feature
class AdvAdjExtractor(BaseEstimator, TransformerMixin):

    def starting_verb(self, text):
        token_list = tokenize(text)
        pos_tags = nltk.pos_tag(token_list)
        adj_adv = 0
        for word, tag in pos_tags:
            if tag in ['JJ','JJR','JJS','RB','RBR','RBS']:
                adj_adv = adj_adv+1
        return adj_adv

    def fit(self, x, y=None):
        return self

    def transform(self, X):
        X_tagged = pd.Series(X).apply(self.starting_verb)
        return pd.DataFrame(X_tagged)
# Add number of adj and adv count as new feature and try again
pipeline2 = Pipeline([
    ('features',FeatureUnion([
        ('text_pipeline',Pipeline([
            ('vect', CountVectorizer(tokenizer=tokenize)),
            ('tfidf', TfidfTransformer())
        ])),
        ('adv_adj', AdvAdjExtractor())
    ])),
    ('clf', MultiOutputClassifier(RandomForestClassifier()))
])

parameters2 = {
    'clf__estimator__n_estimators': [10,20]
}

cv2 = GridSearchCV(pipeline2, param_grid=parameters2,n_jobs=10)
cv2.fit(X_train, y_train)
y_pred = cv2.predict(X_test)
# Evaluate improvement result
df_pred3 = pd.DataFrame(y_pred)
df_pred3.columns=y_test.columns

for column in y_test:
    print(column)
    print(classification_report(y_test[column],df_pred3[column]))<jupyter_output>related
             precision    recall  f1-score   support

          0       0.67      0.32      0.43      1487
          1       0.82      0.95      0.88      5018
          2       0.46      0.24      0.32        49

avg / total       0.78      0.80      0.78      6554

request
             precision    recall  f1-score   support

          0       0.89      0.99      0.93      5434
          1       0.87      0.38      0.53      1120

avg / total       0.88      0.88      0.86      6554

offer
             precision    recall  f1-score   support

          0       0.99      1.00      1.00      6518
          1       0.00      0.00      0.00        36

avg / total       0.99      0.99      0.99      6554

aid_related
             precision    recall  f1-score   support

          0       0.74      0.86      0.79      3819
          1       0.75      0.57      0.65      2735

avg / total       0.74      0.74      0.73      6554

medical_help
             precision    recall  f1-sco[...]<jupyter_text>9. Export your model as a pickle file<jupyter_code>pickle.dump(cv2, open('message_classification_model.sav', 'wb'))<jupyter_output><empty_output><jupyter_text>10. Use this notebook to complete `train.py`Use the template file attached in the Resources folder to write a script that runs the steps above to create a database and export a model based on a new dataset specified by the user.<jupyter_code>import nltk
nltk.download(['punkt', 'wordnet','averaged_perceptron_tagger'])

import sys
import re
import pandas as pd
import sqlalchemy as db
from nltk.tokenize import word_tokenize
from nltk.stem import WordNetLemmatizer
import pickle

from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import GridSearchCV
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.multioutput import MultiOutputClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.metrics import classification_report
from sklearn import model_selection
from sklearn.neighbors import KNeighborsClassifier

class AdvAdjExtractor(BaseEstimator, TransformerMixin):

    def starting_verb(self, text):
        token_list = tokenize(text)
        pos_tags = nltk.pos_tag(token_list)
        adj_adv = 0
        for word, tag in pos_tags:
            if tag in ['JJ','JJR','JJS','RB','RBR','RBS']:
                adj_adv = adj_adv+1
        return adj_adv

    def fit(self, x, y=None):
        return self

    def transform(self, X):
        X_tagged = pd.Series(X).apply(self.starting_verb)
        return pd.DataFrame(X_tagged)

def load_data(database_filepath):
    engine = db.create_engine('sqlite:///'+database_filepath)
    df = pd.read_sql_table('CategorisedMessages', engine)
    category_names = ['related', 'request', 'offer',
       'aid_related', 'medical_help', 'medical_products', 'search_and_rescue',
       'security', 'military', 'child_alone', 'water', 'food', 'shelter',
       'clothing', 'money', 'missing_people', 'refugees', 'death', 'other_aid',
       'infrastructure_related', 'transport', 'buildings', 'electricity',
       'tools', 'hospitals', 'shops', 'aid_centers', 'other_infrastructure',
       'weather_related', 'floods', 'storm', 'fire', 'earthquake', 'cold',
       'other_weather', 'direct_report']
    X = df['message']
    Y = df[category_names]
    
    return X, Y, category_names


def tokenize(text):
    # Replace none-character with space
    text = re.sub('[^A-Za-z0-9]',' ',text)
    
    # Tokenize the input
    tokens = word_tokenize(text)
    
    # Initialize lemmatizer for standardize form of words
    lemmatizer = WordNetLemmatizer()
    
    # We will iterate each token in the list, lemmatize and return result
    clean_tokens = []
    for tok in tokens:
        clean_tok = lemmatizer.lemmatize(tok).lower().strip()
        clean_tokens.append(clean_tok)

    return clean_tokens


def build_model():
    # Add number of adj and adv count as new feature and try again
    pipeline = Pipeline([
        ('features',FeatureUnion([
            ('text_pipeline',Pipeline([
                ('vect', CountVectorizer(tokenizer=tokenize)),
                ('tfidf', TfidfTransformer())
            ])),
            ('adv_adj', AdvAdjExtractor())
        ])),
        ('clf', MultiOutputClassifier(RandomForestClassifier()))
    ])

    parameters = {
        'clf__estimator__n_estimators': [10,20]
    }

    return GridSearchCV(pipeline, param_grid=parameters,n_jobs=4)
    

def evaluate_model(model, X_test, Y_test, category_names):
    y_pred = model.predict(X_test)
    
    # Convert to dataframe for ease of iteration
    df_y_pred = pd.DataFrame(y_pred)
    df_y_pred.columns=category_names
    
    # Evaluate improvement result
    for column in category_names:
        print(column)
        print(classification_report(Y_test[column],df_y_pred[column]))


def save_model(model, model_filepath):
    pickle.dump(model, open(model_filepath, 'wb'))


def main():
    if len(sys.argv) == 3:
        database_filepath, model_filepath = sys.argv[1:]
        
        print('Loading data...\n    DATABASE: {}'.format(database_filepath))
        X, Y, category_names = load_data(database_filepath)
        X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.2)
        
        print('Building model...')
        model = build_model()
        
        print('Training model...')
        model.fit(X_train, Y_train)
        
        print('Evaluating model...')
        evaluate_model(model, X_test, Y_test, category_names)

        print('Saving model...\n    MODEL: {}'.format(model_filepath))
        save_model(model, model_filepath)

        print('Trained model saved!')

    else:
        print('Please provide the filepath of the disaster messages database '\
              'as the first argument and the filepath of the pickle file to '\
              'save the model to as the second argument. \n\nExample: python '\
              'train_classifier.py ../data/DisasterResponse.db classifier.pkl')


if __name__ == '__main__':
    main()<jupyter_output>[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package wordnet to /root/nltk_data...
[nltk_data]   Package wordnet is already up-to-date!
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /root/nltk_data...
[nltk_data]   Package averaged_perceptron_tagger is already up-to-
[nltk_data]       date!
Loading data...
    DATABASE: InsertDatabaseName.db
Building model...
Training model...
Evaluating model...
related
             precision    recall  f1-score   support

          0       0.68      0.32      0.43      1237
          1       0.81      0.95      0.88      3970
          2       0.62      0.28      0.38        36

avg / total       0.78      0.80      0.77      5243

request
             precision    recall  f1-score   support

          0       0.89      0.99      0.94      4355
          1       0.87      0.43      0.57       888

avg / total       0.89      0.89 [...]
{'fraction_non_alphanumeric': 0.09461036731891521, 'fraction_numerical': 0.034466186062478546, 'mean_word_length': 2.7843595739153026, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 38, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 28, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '273700', 'license': 'MIT', 'n_tokens_mistral': 4809, 'n_tokens_neox': 4528, 'n_words': 1204, 'path': 'Jupyter Notebook/ML Pipeline Preparation.ipynb', 'repo_name': 'pathompong-y/DisasterMsgClassification', 'size': '14565'}