<jupyter_start><jupyter_text>ML Pipeline PreparationFollow the instructions below to help you create your ML pipeline. 1. Import libraries and load data from database.- Import Python libraries- Load dataset from database with [`read_sql_table`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_sql_table.html)- Define feature and target variables X and Y<jupyter_code># import libraries
import pandas as pd
import numpy as np
import re
from time import time
from sqlalchemy import create_engine
import nltk
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem.wordnet import WordNetLemmatizer
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer 
from sklearn.multioutput import MultiOutputClassifier
from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.metrics import classification_report, accuracy_score, precision_score, recall_score
import pickle
nltk.download(['punkt', 'wordnet', 'stopwords'])
from collections import Counter
# load data from database
engine = create_engine('sqlite:///../data/DisasterResponse.db')
table_name = "messages"
df = pd.read_sql_table(table_name, engine)
X = df.message
Y = df.drop(['id','message','genre','original'], axis=1)
X.head()
Y.head()<jupyter_output><empty_output><jupyter_text>2. Write a tokenization function to process your text data<jupyter_code>def tokenize(text):
    """
    Function: tokenize text
    Args:
        text (str): string to tokenize
    Return:
        clean_tokens (list of str): list of cleaned and tokenized strings
    """
    # 1. normalization
    # remove punctuation and capitalization of words
    text = re.sub(r'[^a-zA-Z0-9]', " ", text.lower())
    
    # 2. tokenization
    # tokenize words from text
    tokens = word_tokenize(text)
    
    # 3. stop word removal
    # remove all words that can be found in the stopwords dictionary
    tokens = [w for w in tokens if w not in stopwords.words("english")]

    # 4. stemming and lemmtization
    # define lemmatizer
    lemmatizer = WordNetLemmatizer()
    
    # iterate through each token
    clean_tokens = []
    for tok in tokens:
        # lemmatize and remove leading/trailing white space
        clean_tok = lemmatizer.lemmatize(tok).strip()
        clean_tokens.append(clean_tok)

    return clean_tokens
# check funtion
print("Message: ", X[0])
print("Tokens: ", tokenize(X[0]))<jupyter_output>Message:  Weather update - a cold front from Cuba that could pass over Haiti
Tokens:  ['weather', 'update', 'cold', 'front', 'cuba', 'could', 'pas', 'haiti']<jupyter_text>3. Build a machine learning pipelineThis machine pipeline should take in the `message` column as input and output classification results on the other 36 categories in the dataset. You may find the [MultiOutputClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.multioutput.MultiOutputClassifier.html) helpful for predicting multiple target variables.<jupyter_code># define ML pipeline with TF-IDF Vectorizer and MultiOutputClassifier (as estimator we use a RandomForestClassifier)
pipeline1 = Pipeline([
    ('vect', TfidfVectorizer(tokenizer=tokenize)),
    ('clf', MultiOutputClassifier(RandomForestClassifier(n_estimators=100)))    
])<jupyter_output><empty_output><jupyter_text>4. Train pipeline- Split data into train and test sets- Train pipeline<jupyter_code># split dataset into training and testing datasets
X_train, X_test, y_train, y_test = train_test_split(X,Y, test_size = 0.20, random_state=42)

# train pipeline
start = time()
pipeline1.fit(X_train, y_train)
print("Training Time: {:.0f} seconds".format(time()-start))<jupyter_output>Training Time: 447 seconds<jupyter_text>5. Test your modelReport the f1 score, precision and recall for each output category of the dataset. You can do this by iterating through the columns and calling sklearn's `classification_report` on each.<jupyter_code>def display_accuracy(y_test, y_pred):
    """
    Function: show accuracy metrics for predicted test dataset (f1_score, precision, recall, support)
    Args:
        y_test (pd.Series): original labels of test dataset
        y_pred (pd.Series): predicted labels of test dataset
    Return:
        None
    """
    for i, col in enumerate(y_test):
        print(col,':')
        print(classification_report(y_test[col], y_pred[:, i]))
y_pred1 = pipeline1.predict(X_test)
display_accuracy(y_test, y_pred1)
# print overall accuracy, precision and recall of tuned model
print('accuracy:\t{:.4}'.format(accuracy_score(y_test,y_pred)))
print('precision:\t{:.4}'.format(precision_score(y_test,y_pred, average='weighted')))
print('recall:\t\t{:.4}'.format(recall_score(y_test,y_pred, average='weighted')))<jupyter_output>accuracy:	0.2785
precision:	0.7654
recall:		0.5345<jupyter_text>6. Improve your modelUse grid search to find better parameters.<jupyter_code># inspect pipeline parameters
pipeline1.get_params(deep=True)
# define selection of parameters for GridSearch
# we limit ourselves to only three parameters since the training time will massively increase

parameters = {
    'vect__ngram_range': [(1,1),(1,2)],
    'clf__estimator__n_estimators': [50, 100],
    'clf__estimator__min_samples_split': [2, 4]
}

cv1 = GridSearchCV(pipeline, param_grid=parameters, cv=3, n_jobs=1)
# train GridSearch object
start = time()
cv1.fit(X_train, y_train)
print("Training Time: {:.0f} seconds".format(time()-start))<jupyter_output>Training Time: 9474 seconds<jupyter_text>7. Test your modelShow the accuracy, precision, and recall of the tuned model.  Since this project focuses on code quality, process, and  pipelines, there is no minimum performance metric needed to pass. However, make sure to fine tune your models for accuracy, precision and recall to make your project stand out - especially for your portfolio!<jupyter_code># predict output of test dataset with tuned model
y_pred_cv1 = cv1.best_estimator_.predict(X_test)
print(classification_report(y_test, y_pred_cv1, target_names=list(y_test.columns)))
# print overall accuracy, precision and recall of tuned model
print('accuracy:\t{:.4}'.format(accuracy_score(y_test,y_pred_cv1)))
print('precision:\t{:.4}'.format(precision_score(y_test,y_pred_cv1, average='weighted')))
print('recall:\t\t{:.4}'.format(recall_score(y_test,y_pred_cv1, average='weighted')))
# print parameters of tuned model
cv1.best_estimator_.get_params()<jupyter_output><empty_output><jupyter_text>8. Try improving your model further. Here are a few ideas:* try other machine learning algorithms* add other features besides the TF-IDF<jupyter_code># define new pipeline with another ML algorithm
pipeline2 = Pipeline([
    ('vect', TfidfVectorizer(tokenizer=tokenize)),
    ('clf', MultiOutputClassifier(AdaBoostClassifier()))    
])
# train pipeline
start = time()
pipeline2.fit(X_train, y_train)
print("Training Time: {:.0f} seconds".format(time()-start))
y_pred2 = pipeline2.predict(X_test)
display_accuracy(y_test, y_pred2)
# print overall accuracy, precision and recall of tuned model
print('accuracy:\t{:.4}'.format(accuracy_score(y_test,y_pred2)))
print('precision:\t{:.4}'.format(precision_score(y_test,y_pred2, average='weighted')))
print('recall:\t\t{:.4}'.format(recall_score(y_test,y_pred2, average='weighted')))
# inspect pipeline parameters
pipeline2.get_params(deep=True)
# define selection of parameters for GridSearch
parameters = {
    'vect__max_df': [0.9, 1.0],
    'clf__estimator__n_estimators': [50, 100],
    'clf__estimator__learning_rate':[0.9, 1.0]
}

cv2 = GridSearchCV(pipeline2, param_grid=parameters, cv=3, n_jobs=1)
# train new GridSearch object
start = time()
cv2.fit(X_train, y_train)
print("Training Time: {:.0f} seconds".format(time()-start))
# predict output of test dataset with new tuned model
y_pred_cv2 = cv2.best_estimator_.predict(X_test)
print(classification_report(y_test, y_pred_cv2, target_names=list(y_test.columns)))
# print overall accuracy, precision and recall of tuned model
print('accuracy:\t{:.4}'.format(accuracy_score(y_test,y_pred_cv2)))
print('precision:\t{:.4}'.format(precision_score(y_test,y_pred_cv2, average='weighted')))
print('recall:\t\t{:.4}'.format(recall_score(y_test,y_pred_cv2, average='weighted')))
# print parameters of tuned model
cv2.best_estimator_.get_params()<jupyter_output><empty_output><jupyter_text>9. Export your model as a pickle file<jupyter_code># since the f1_score of the second model tuned with GridSearch is better than the first one,
# we will export cv2
with open('../models/disaster_response_clf.pkl', 'wb') as f:
    pickle.dump(cv2, f)<jupyter_output><empty_output><jupyter_text>10. Use this notebook to complete `train.py`Use the template file attached in the Resources folder to write a script that runs the steps above to create a database and export a model based on a new dataset specified by the user.<jupyter_code># import libraries
import pandas as pd
import numpy as np
import re
from time import time
from sqlalchemy import create_engine
import nltk
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.stem.wordnet import WordNetLemmatizer
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer 
from sklearn.multioutput import MultiOutputClassifier
from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.metrics import classification_report, accuracy_score, precision_score, recall_score
nltk.download(['punkt', 'wordnet', 'stopwords'])

def load_data(db_path = '../data/DisasterResponse.db', table_name = "messages"):
    '''
    Function:   load data from SQLite database and return X and y variables
    Args:
        data_file (str):    path of SQLite database
        table_name (str):   name of table in SQLite database
    Return:
        X (pd.DataFrame):   messages (features)
        y (pd.DataFrame):   categories (labels)
    '''
    engine = create_engine('sqlite:///'+db_path)
    df = pd.read_sql_table(table_name, engine)
    X = df.message
    y = df.drop(['id','message','genre','original'], axis=1)
    return X, y

def tokenize(text):
    """
    Function: tokenize text
    Args:
        text (str): string to tokenize
    Return:
        clean_tokens (list of str): list of cleaned and tokenized strings
    """
    # 1. normalization
    # remove punctuation and capitalization of words
    text = re.sub(r'[^a-zA-Z0-9]', " ", text.lower())
    
    # 2. tokenization
    # tokenize words from text
    tokens = word_tokenize(text)
    
    # 3. stop word removal
    # remove all words that can be found in the stopwords dictionary
    tokens = [w for w in tokens if w not in stopwords.words("english")]

    # 4. stemming and lemmtization
    # define lemmatizer
    lemmatizer = WordNetLemmatizer()
    
    # iterate through each token
    clean_tokens = []
    for tok in tokens:
        # lemmatize and remove leading/trailing white space
        clean_tok = lemmatizer.lemmatize(tok).strip()
        clean_tokens.append(clean_tok)

    return clean_tokens

def build_model():
    """
    Function: create GridSearch model consisting of TF-IDF-Vectorizer and AdaBoostClassifier
    Args:
        None
    Return:
        cv (GridSearch): GridSearchObject
    """
    # create pipeline
    pipeline = Pipeline([
        ('vect', TfidfVectorizer(tokenizer=tokenize)),
        ('clf', MultiOutputClassifier(AdaBoostClassifier()))
    ])
    
    # define parameters for GridSearch
    parameters = {
        'vect__max_df': [0.9, 1.0],
        'clf__estimator__n_estimators': [50, 100],
        'clf__estimator__learning_rate':[0.9, 1.0]
    }
    
    # create model
    cv = GridSearchCV(pipeline, param_grid=parameters, cv=3, n_jobs=1)
    
    return cv

def evaluate_model(model, X_test, y_test):
    """
    Function: predict output of model and evaluates the result against the testing dataset
    Args:
        model (classifier obj): trained classifier model
        X_test (pd.DataFrame): features of testing dataset
        y_test (pd.DataFrame): labels of testing dataset
    Return:
       None
    """
    # make predictions
    y_pred = model.predict(X_test)
    
    # create classification report
    print(classification_report(y_test, y_pred, target_names=list(y_test.columns)))
    
    # print total accuracy, precision, recall on testing dataset
    print('accuracy:\t{:.4}'.format(accuracy_score(y_test,y_pred)))
    print('precision:\t{:.4}'.format(precision_score(y_test,y_pred, average='weighted')))
    print('recall:\t\t{:.4}'.format(recall_score(y_test,y_pred, average='weighted')))
    
def export_model(model, clf_path = '../models/disaster_response_clf.pkl'):
    """
    Function: save model as pickle file
    Args:
        model (GridSearch obj): trained and tuned classifier model
        clf_path (str): path of pickle file destination
    Return:
        None
    """
    with open(clf_path, 'wb') as f:
        pickle.dump(model, f)

        
def import_model(clf_path = '../models/disaster_response_clf.pkl'):
    """
    Function: load model from pickle file
    Args:
        clf_path (str): path of pickle file
    Return:
        model (GridSearch obj): loaded model
    """
    with open(clf_path, 'rb') as f:
        model = pickle.load(f)
    
    return model

def main():
    print("Loading data...")
    X,y = load_data()
    print("Data loaded!")
    print("------------")
    
    print("Building model...")
    model = build_model()
    print("Model built!")
    print("------------")
    
    print("Training model...")
    start = time()
    X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.20, random_state=42)
    model.fit(X_train, y_train)
    print("Training Time: {:.0f} seconds".format(time()-start))
    print("Model trained!")
    print("------------")
    
    print("Evaluating model...")
    evaluate_model(model, X_test, y_test)
    print("Model evaluated!")
    print("------------")
    
    print("Saving model...")
    clf_path = '../models/disaster_response_clf1.pkl'
    export_model(model.best_estimator_, clf_path)
    print("Model saved at ", clf_path, "!")
    print("------------")

if __name__== "__main__":
    main()
    
    
# TEST: function for counting words to create a visualization with the top 25 words within the web application

def count_words(df):
    '''
    Function: counts all tokens and returns the top 25 words
    Args:
        df (pd.DataFrame): DataFrame containing all messages
    Return:
        top_words (Counter): top 25 words and their frequency
    '''
    # create Counter object
    words = Counter()
    # iterate through messages in DataFrame
    for message in df.message:
        # tokenize each message
        tokens = tokenize(message)
        # iterate through created tokens
        for token in tokens:
            words[token] += 1
    # filter top 25 words
    top_words = words.most_common(25)
    return top_words 
# get most frequent 25 words
start = time()
top_words = count_words(df)
print("{:.0f} seconds".format(time()-start))
top_words
# save most frequent 25 words and their frequency in text file
with open('../data/top_words.txt', 'w') as f:
    for item in top_words:
        f.write("{}\n".format(item))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1111111111111111, 'fraction_numerical': 0.011842360706658901, 'mean_word_length': 4.213900134952767, 'pattern_counts': {'":': 1, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '560530', 'license': 'CNRI-Python', 'n_tokens_mistral': 4963, 'n_tokens_neox': 4785, 'n_words': 1369, 'path': 'notebooks/ML Pipeline Preparation.ipynb', 'repo_name': 'oliverkroening/Udacity_DSND_Project05', 'size': '15453'}