<jupyter_start><jupyter_text>ML Pipeline PreparationFollow the instructions below to help you create your ML pipeline. 1. Import libraries and load data from database.- Import Python libraries- Load dataset from database with [`read_sql_table`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_sql_table.html)- Define feature and target variables X and Y<jupyter_code># import libraries
import re
import numpy as np
import pandas as pd
from nltk.tokenize import word_tokenize
from nltk.stem import WordNetLemmatizer
from nltk.corpus import stopwords
from sqlalchemy import create_engine
from nltk.stem.porter import PorterStemmer

from sklearn.metrics import confusion_matrix
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split,GridSearchCV
from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
import nltk
nltk.download(['punkt', 'wordnet', 'averaged_perceptron_tagger'])

# load data from database
DATABASE_FILENAME = '../db.sqlite3'
TABLE_NAME = 'disaster_message'
engine = create_engine('sqlite:///' + DATABASE_FILENAME)
df = pd.read_sql_table(TABLE_NAME, engine)
X = df['message']
Y = df.iloc[:, 4:]
category_names = list(df.columns[4:])
X
df.columns[4:]<jupyter_output><empty_output><jupyter_text>2. Write a tokenization function to process your text data<jupyter_code>def tokenize(text):
    # normalize text and remove punctuation
    text = re.sub(r"[^a-zA-Z0-9]", " ", text.lower())
    
    # tokenize text
    tokens = word_tokenize(text)
    
     # initiate lemmatizer
    lemmatizer = WordNetLemmatizer()

    # iterate through each token
    clean_tokens = []
    for tok in tokens:
        # lemmatize, normalize case, and remove leading/trailing white space
        clean_tok = lemmatizer.lemmatize(tok).lower().strip()
        clean_tokens.append(clean_tok)

    return clean_tokens<jupyter_output><empty_output><jupyter_text>3. Build a machine learning pipelineThis machine pipeline should take in the `message` column as input and output classification results on the other 36 categories in the dataset. You may find the [MultiOutputClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.multioutput.MultiOutputClassifier.html) helpful for predicting multiple target variables.<jupyter_code>from sklearn.multioutput import MultiOutputClassifier
pipeline = Pipeline([
    ('vect',CountVectorizer(tokenizer=tokenize)),
    ('tfidf', TfidfTransformer()),
    ('clf',MultiOutputClassifier(RandomForestClassifier(n_jobs=-1))),
])<jupyter_output><empty_output><jupyter_text>4. Train pipeline- Split data into train and test sets- Train pipeline<jupyter_code>X_train,X_test,y_train,y_test = train_test_split(X,Y)<jupyter_output><empty_output><jupyter_text>5. Test your modelReport the f1 score, precision and recall for each output category of the dataset. You can do this by iterating through the columns and calling sklearn's `classification_report` on each.<jupyter_code>%%time
pipeline.fit(X_train,y_train)
%%time
y_pred = pipeline.predict(X_test)
from sklearn.metrics import classification_report,accuracy_score
y_pred.shape,y_test.shape,len(list(Y.columns))
print(classification_report(y_test.iloc[:, 1:].values, np.array([x[1:] for x in y_pred])))<jupyter_output>precision    recall  f1-score   support

           0       0.88      0.45      0.60      1131
           1       0.00      0.00      0.00        33
           2       0.79      0.62      0.70      2722
           3       0.65      0.06      0.11       540
           4       0.80      0.07      0.13       328
           5       0.86      0.03      0.07       172
           6       0.00      0.00      0.00       120
           7       0.63      0.05      0.10       232
           8       0.00      0.00      0.00         0
           9       0.93      0.23      0.37       427
          10       0.87      0.48      0.62       729
          11       0.85      0.24      0.38       596
          12       0.86      0.07      0.14        80
          13       1.00      0.04      0.07       171
          14       1.00      0.01      0.03        75
          15       0.75      0.01      0.03       206
          16       0.90      0.12      0.21       298
          17       0.47    [...]<jupyter_text>6. Improve your modelUse grid search to find better parameters.<jupyter_code>pipeline.get_params()
parameters = {
    'vect__max_df': (0.5, 0.75, 1.0),
    'vect__ngram_range': ((1, 1), (1,2)),
    'vect__stop_words':(None,'english'),
    'vect__max_features': (None, 5000,10000),
    'tfidf__use_idf': (True, False)
}
cv = GridSearchCV(pipeline, param_grid=parameters)<jupyter_output>Wall time: 80.6 ms<jupyter_text>7. Test your modelShow the accuracy, precision, and recall of the tuned model.  Since this project focuses on code quality, process, and  pipelines, there is no minimum performance metric needed to pass. However, make sure to fine tune your models for accuracy, precision and recall to make your project stand out - especially for your portfolio!<jupyter_code>%%time
cv.fit(X_train,y_train)
print('Best Parameters:', cv.best_params_)
#Best Parameters: {'tfidf__use_idf': True, 'vect__max_df': 0.75, 'vect__max_features': 5000, 'vect__ngram_range': (1, 2), 'vect__stop_words': 'english'}
#Wall time: 10h 54min 48s
%%time
y_pred = cv.predict(X_test)

len(y_pred)
print(classification_report(y_test.iloc[:, 1:].values, np.array([x[1:] for x in y_pred])))<jupyter_output>precision    recall  f1-score   support

           0       0.79      0.50      0.61      1131
           1       0.00      0.00      0.00        33
           2       0.74      0.70      0.72      2722
           3       0.63      0.16      0.25       540
           4       0.75      0.20      0.31       328
           5       0.74      0.13      0.23       172
           6       0.17      0.01      0.02       120
           7       0.55      0.11      0.19       232
           8       0.00      0.00      0.00         0
           9       0.85      0.55      0.67       427
          10       0.80      0.74      0.77       729
          11       0.81      0.50      0.62       596
          12       0.83      0.24      0.37        80
          13       0.86      0.04      0.07       171
          14       1.00      0.03      0.05        75
          15       0.58      0.17      0.26       206
          16       0.76      0.34      0.47       298
          17       0.61    [...]<jupyter_text>8. Try improving your model further. Here are a few ideas:* try other machine learning algorithms* add other features besides the TF-IDF Randomforest takes forever to run, so from (https://scikit-learn.org/stable/modules/sgd.html) I decide to use another ML method called Stochastic Gradient Descent (SGD)    other feature could be like hashvectorizer, but it was running for too long, so I gave up and tried another two ML Algorithms, MultinomialNB and AdanboostClassifier, so far AdaboostClassifier's performace is alomost like random forest and with way less run time<jupyter_code>from sklearn.naive_bayes import MultinomialNB
pipeline_MNB = Pipeline([
    ('vect',CountVectorizer(tokenizer=tokenize)),
    ('tfidf', TfidfTransformer()),
    ('clf',MultiOutputClassifier(MultinomialNB())),
])
%%time
pipeline_MNB.fit(X_train,y_train)
y_pred_MNB = pipeline_MNB.predict(X_test)
y_pred_MNB.shape,y_test.shape,len(list(Y.columns))
print(classification_report(y_test.iloc[:, 1:].values, np.array([x[1:] for x in y_pred_MNB])))
pipeline_Ada = Pipeline([
    ('vect',CountVectorizer(tokenizer=tokenize)),
    ('tfidf', TfidfTransformer()),
    ('clf',MultiOutputClassifier(AdaBoostClassifier()))
])
%%time
pipeline_Ada.fit(X_train,y_train)
y_pred_ada = pipeline_Ada.predict(X_test)
print(classification_report(y_test.iloc[:, 1:].values, np.array([x[1:] for x in y_pred_ada])))
pipeline_Ada.get_params()
parameters_Ada = {
    'vect__stop_words': (None,True),
    'vect__max_features': (None, 5000,10000),
    'tfidf__use_idf': (True, False)
}

cv_Ada = GridSearchCV(pipeline_Ada, param_grid=parameters_Ada)
%%time
cv_Ada.fit(X_train,y_train)
print('Best Parameters:', cv_Ada.best_params_)
#Best Parameters: {'tfidf__use_idf': True, 'vect__max_features': 10000, 'vect__stop_words': None}
#Wall time: 33min 31s
%%time
y_pred_cv_ada = cv_Ada.predict(X_test)
print(classification_report(y_test.iloc[:, 1:].values, np.array([x[1:] for x in y_pred_cv_ada])))<jupyter_output>precision    recall  f1-score   support

           0       0.77      0.55      0.64      1131
           1       0.00      0.00      0.00        33
           2       0.76      0.60      0.67      2722
           3       0.66      0.27      0.39       540
           4       0.66      0.32      0.43       328
           5       0.60      0.16      0.26       172
           6       0.23      0.04      0.07       120
           7       0.58      0.31      0.41       232
           8       0.00      0.00      0.00         0
           9       0.73      0.59      0.65       427
          10       0.80      0.68      0.73       729
          11       0.77      0.55      0.64       596
          12       0.76      0.51      0.61        80
          13       0.64      0.22      0.32       171
          14       0.73      0.21      0.33        75
          15       0.59      0.25      0.35       206
          16       0.65      0.41      0.50       298
          17       0.52    [...]<jupyter_text>9. Export your model as a pickle file<jupyter_code>import joblib
#save best parm model from random forest model.
joblib.dump(cv, 'randomF.pkl')
joblib.dump(cv.best_estimator_, 'randomF_best.pkl')
#save ada model with the best estimator
joblib.dump(cv_Ada, 'Ada.pkl')
joblib.dump(cv_Ada.best_estimator_, 'Ada_best.pkl')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09472409966710381, 'fraction_numerical': 0.0784828003631595, 'mean_word_length': 2.1775641025641024, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 52, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '231390', 'license': 'FTL', 'n_tokens_mistral': 3630, 'n_tokens_neox': 3242, 'n_words': 862, 'path': 'ML Pipeline Preparation.ipynb', 'repo_name': 'boniu86/Disaster_Response_Pipeline', 'size': '9913'}