<jupyter_start><jupyter_text>ML Pipeline PreparationFollow the instructions below to help you create your ML pipeline. 1. Import libraries and load data from database.- Import Python libraries- Load dataset from database with [`read_sql_table`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_sql_table.html)- Define feature and target variables X and Y<jupyter_code># import libraries
import time
import re
import nltk
import pickle
import pandas as pd

from sqlalchemy import create_engine
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords, wordnet
from nltk.stem.wordnet import WordNetLemmatizer
from nltk.stem.porter import PorterStemmer
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.pipeline import Pipeline
from sklearn.multioutput import MultiOutputClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.metrics import classification_report
from sklearn.naive_bayes import MultinomialNB
from sklearn.svm import SVC

nltk.download('punkt')
nltk.download('stopwords')
nltk.download('wordnet')
nltk.download('averaged_perceptron_tagger')

# Suppress warnings
import warnings; warnings.simplefilter('ignore')
# load data from database
engine = create_engine('sqlite:///Messages.db')
df = pd.read_sql_table('Messages', engine) 
# Print first few lines of dataframe
df.head(2)
# Define X and Y variables
X = df['message']
Y = df.drop(['id', 'message', 'original', 'genre'], axis=1)<jupyter_output><empty_output><jupyter_text>2. Write a tokenization function to process your text data<jupyter_code># Define tokenize function
# Excluded the lemmatization step to increase speed
# Defining the variable "stop_words" makes the code run significantly faster

def tokenize(text):
    
    # Case normalization
    text = text.lower()
    
    # Puncuation removal
    text = re.sub(r'[^a-zA-Z0-9]', ' ', text)
    
    # Tokenize words
    words = word_tokenize(text)
    
    # Stop word removal
    stop_words = stopwords.words("english")
    words = [w for w in words if w not in stop_words]
    
    # Perform stemming
    stemmer = PorterStemmer()
    words = [stemmer.stem(w) for w in words if w not in stop_words]
    
    return words
%%timeit

# Time how long the tokenize function to work on this short sentence
# to help with speed optimization

text = "You are the greatest, no 1.0, person in the world."
tokenize(text)<jupyter_output>962 µs ± 8.54 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)<jupyter_text>3. Build a machine learning pipelineThis machine pipeline should take in the `message` column as input and output classification results on the other 36 categories in the dataset. You may find the [MultiOutputClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.multioutput.MultiOutputClassifier.html) helpful for predicting multiple target variables.<jupyter_code># Build ML pipeline using random forest classifier
pipeline1 = Pipeline([
    ('vect', CountVectorizer(tokenizer = tokenize)),
    ('tfidf', TfidfTransformer()),
    ('clf', MultiOutputClassifier(RandomForestClassifier()))
])<jupyter_output><empty_output><jupyter_text>4. Train pipeline- Split data into train and test sets- Train pipeline<jupyter_code># Split data into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, Y)

start = time.time()

# Train model
pipeline1.fit(X_train, y_train)

end = time.time()
train_time1 = end - start
print(train_time1)<jupyter_output>96.1573212146759<jupyter_text>5. Test your modelReport the f1 score, precision and recall for each output category of the dataset. You can do this by iterating through the columns and calling sklearn's `classification_report` on each.<jupyter_code># Predict labels using model
start = time.time()

y_pred1 = pipeline1.predict(X_test)

end = time.time()
pred_time1 = end - start
print(pred_time1)
# Print accuracy report 
report1 = pd.DataFrame.from_dict(classification_report(y_test, y_pred1, target_names=Y.columns, output_dict=True))
report1 = pd.DataFrame.transpose(report1)
report1<jupyter_output><empty_output><jupyter_text>6. Improve your modelUse grid search to find better parameters.<jupyter_code># Use grid search to find better parameters

start = time.time()

parameters = {
    'clf__estimator__n_estimators': [10, 100],
    'clf__estimator__min_samples_split': [2, 5]
}

cv = GridSearchCV(pipeline1, param_grid=parameters, verbose=1)
cv.fit(X_train, y_train)
y_pred_cv = cv.predict(X_test)

end = time.time()
cv_time = end - start
print(cv_time)
# Show results of gridsearch.  
# According to the rank_test_score and train_scores:
# param_clf__estimator__min_samples_split = 2 and param_clf__estimator__n_estimators = 100
# is the best option (also the slowest).

pd.DataFrame.from_dict(cv.cv_results_)<jupyter_output><empty_output><jupyter_text>7. Test your modelShow the accuracy, precision, and recall of the tuned model.  Since this project focuses on code quality, process, and  pipelines, there is no minimum performance metric needed to pass. However, make sure to fine tune your models for accuracy, precision and recall to make your project stand out - especially for your portfolio!<jupyter_code># Print accuracy report 
report_cv = pd.DataFrame.from_dict(classification_report(y_test, y_pred_cv, target_names=Y.columns, output_dict=True))
report_cv = pd.DataFrame.transpose(report_cv)
report_cv
# Compare f1 scores between default model and improved model
report_cv['f1-score'] - report1['f1-score']<jupyter_output><empty_output><jupyter_text>The default RF and the improved model appear to be similar in terms of f1 score.  8. Try improving your model further. Here are a few ideas:* try other machine learning algorithms* add other features besides the TF-IDF<jupyter_code># Create new pipeline using MultinomialNB classifier
pipeline2 = Pipeline([
    ('vect', CountVectorizer(tokenizer = tokenize)),
    ('tfidf', TfidfTransformer()),
    ('clf', MultiOutputClassifier(MultinomialNB()))
])
# Split data into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, Y)

start = time.time()

# Train model
pipeline2.fit(X_train, y_train)

end = time.time()
train_time2 = end - start
print(train_time2)
# Test model
start = time.time()

y_pred2 = pipeline2.predict(X_test)

end = time.time()
pred_time2 = end - start
print(pred_time2)
# Print accuracy report 
report2 = pd.DataFrame.from_dict(classification_report(y_test, y_pred2, target_names=Y.columns, output_dict=True))
report2 = pd.DataFrame.transpose(report2)
report2<jupyter_output><empty_output><jupyter_text>The MultinomialNB classifier performs poorly compared to the RandomForestClassifier as it cannot predict many of the categories.<jupyter_code># Create new pipeline using SVC classifier
pipeline3 = Pipeline([
    ('vect', CountVectorizer(tokenizer = tokenize)),
    ('tfidf', TfidfTransformer()),
    ('clf', MultiOutputClassifier(SVC()))
])
# Split data into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, Y)

start = time.time()

# Train model
pipeline3.fit(X_train, y_train)

end = time.time()
train_time3 = end - start
print(train_time3)
# Test model
start = time.time()

y_pred3 = pipeline3.predict(X_test)

end = time.time()
pred_time3 = end - start
print(pred_time3)
# Print accuracy report 
report3 = pd.DataFrame.from_dict(classification_report(y_test, y_pred3, target_names=Y.columns, output_dict=True))
report3 = pd.DataFrame.transpose(report3)
report3<jupyter_output><empty_output><jupyter_text>The SVC classifier performs poorly compared to the RandomForestClassifier as it cannot predict anything other than 'related'.  9. Export your model as a pickle file<jupyter_code>pickle.dump(cv, open('disaster_model.sav', 'wb'))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09019408854497019, 'fraction_numerical': 0.012812381073195484, 'mean_word_length': 5.726962457337884, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 39, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '495388', 'license': 'FTL', 'n_tokens_mistral': 2563, 'n_tokens_neox': 2485, 'n_words': 760, 'path': 'ML_Pipeline.ipynb', 'repo_name': 'ursulahneumann/Disaster_Response_Pipeline', 'size': '7883'}