<jupyter_start><jupyter_text>ML Pipeline PreparationFollow the instructions below to help you create your ML pipeline. 1. Import libraries and load data from database.- Import Python libraries- Load dataset from database with [`read_sql_table`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_sql_table.html)- Define feature and target variables X and Y<jupyter_code># import libraries
import pandas as pd
import re
from sqlalchemy import create_engine

from nltk import word_tokenize, sent_tokenize
from nltk.corpus import stopwords
from nltk.stem.wordnet import WordNetLemmatizer

import nltk
nltk.download('punkt')
nltk.download('stopwords')
nltk.download('wordnet')

from sklearn.multioutput import MultiOutputClassifier, ClassifierChain
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB, MultinomialNB
from sklearn.svm import LinearSVC

from sklearn.metrics import confusion_matrix, coverage_error, f1_score, precision_score, recall_score, classification_report, accuracy_score
from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer

from context import haystack
from haystack.modeling.custom_features import QuestionMarkFeature, TextLengthFeature
# load data from database
engine = create_engine('sqlite:///../data/DisasterResponse.db')
df = pd.read_sql_table('categorized_messages', engine)
df.head()
X = df['message']
target_cols = [col for col in df.columns if col not in ('id', 'message', 'original', 'original_language', 'genre')]
Y = df[target_cols]
for col in Y.columns:
    print()
    print(Y[col].value_counts())<jupyter_output>1    19874
0     6115
Name: related, dtype: int64

0    21525
1     4464
Name: request, dtype: int64

0    25872
1      117
Name: offer, dtype: int64

0    15149
1    10840
Name: aid_related, dtype: int64

0    23909
1     2080
Name: medical_help, dtype: int64

0    24679
1     1310
Name: medical_products, dtype: int64

0    25265
1      724
Name: search_and_rescue, dtype: int64

0    25518
1      471
Name: security, dtype: int64

0    25130
1      859
Name: military, dtype: int64

0    25989
Name: child_alone, dtype: int64

0    24320
1     1669
Name: water, dtype: int64

0    23072
1     2917
Name: food, dtype: int64

0    23681
1     2308
Name: shelter, dtype: int64

0    25585
1      404
Name: clothing, dtype: int64

0    25386
1      603
Name: money, dtype: int64

0    25691
1      298
Name: missing_people, dtype: int64

0    25115
1      874
Name: refugees, dtype: int64

0    24797
1     1192
Name: death, dtype: int64

0    22548
1     3441
Name: other_aid, dtype: int64

0    24[...]<jupyter_text>2. Write a tokenization function to process your text data<jupyter_code>def tokenize(text : str) -> List[str]:
    """Custom tokenization function. Normalizes, removes urls and stopwords and lemmatizes the input `text`"""
    
    lemmatizer = WordNetLemmatizer()
    stop_words = stopwords.words('english')
    
    # normalize (remove capitalization)
    text = text.lower()
    
    # replace URLs
    text = re.sub('(http(s)?:\/\/.)?(www\.)?[-a-zA-Z0-9@:%._\+~#=]{2,256}\.[a-z]{2,6}\b([-a-zA-Z0-9@:%_\+.~#?&//=]*)', 'urlplaceholder', text)
    
    # remove punctuation
    text = re.sub('\W', ' ', text)
    
    sentences = sent_tokenize(text)
    
    tokens = []
    for sent in sentences:
        tokens.extend([lemmatizer.lemmatize(word) for word in word_tokenize(sent) if word not in stop_words])
    
    return tokens

test_msg = X.sample(1).iloc[0]
print(test_msg)
tokenize(test_msg)
test_msg = 'How will things be for tomorrow?'
print(test_msg)
tokenize(test_msg)<jupyter_output>How will things be for tomorrow?<jupyter_text>3. Build a machine learning pipelineThis machine pipeline should take in the `message` column as input and output classification results on the other 36 categories in the dataset. You may find the [MultiOutputClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.multioutput.MultiOutputClassifier.html) helpful for predicting multiple target variables.<jupyter_code>pipeline = Pipeline([
    ('vect', CountVectorizer(tokenizer=tokenize)),
    ('tfidf', TfidfTransformer()),
    ('clf', MultiOutputClassifier(RandomForestClassifier()))
])<jupyter_output><empty_output><jupyter_text>4. Train pipeline- Split data into train and test sets- Train pipeline<jupyter_code>seed = 51
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=.3, random_state=seed)
X_train.shape, y_train.shape
max_samples = 250
pipeline.fit(X_train.iloc[:max_samples], y_train.iloc[:max_samples])<jupyter_output><empty_output><jupyter_text>5. Test your modelReport the f1 score, precision and recall for each output category of the dataset. You can do this by iterating through the columns and calling sklearn's `classification_report` on each.<jupyter_code>y_pred = pipeline.predict(X_test)
y_pred = pd.DataFrame(y_pred, columns=y_test.columns)
print(classification_report(y_test, y_pred, target_names=y_test.columns))
for i, col in enumerate(y_test.columns):
        print('*** Category:', col, '***')
        print(classification_report(y_test[col], y_pred[col]))
def eval_metrics_per_category(y_test, y_pred, metrics=[precision_score, recall_score, f1_score, accuracy_score]):
    results = {}
    
    for col in y_test.columns:
        results[col] = {}
        for metric in metrics:
            results[col].update({metric.__name__ : metric(y_test[col], y_pred[col])})
    return pd.DataFrame(results).T

res = eval_metrics_per_category(y_test, y_pred)
res[['precision_score', 'recall_score', 'f1_score']].sort_values('f1_score').plot.bar(figsize=(20, 10))
macro_res = res.mean()
macro_res.plot.bar()<jupyter_output><empty_output><jupyter_text>6. Improve your modelUse grid search to find better parameters.<jupyter_code>parameters = {
    'clf__estimator__max_depth': [3, 5, 8],
    'clf__estimator__n_estimators': [20, 50, 100, 200],
    'clf__estimator': [RandomForestClassifier(), GradientBoostingClassifier()],
    'clf__n_jobs': [-1]
}

cv = GridSearchCV(pipeline, parameters, verbose=4)
cv.fit(X, Y)<jupyter_output>fier\.venv\lib\site-packages\sklearn\utils\validation.py", line 871, in check_X_y
    X = check_array(X, accept_sparse=accept_sparse,
  File "d:\code\udacity\disaster-response-classifier\.venv\lib\site-packages\sklearn\utils\validation.py", line 63, in inner_f
    return f(*args, **kwargs)
  File "d:\code\udacity\disaster-response-classifier\.venv\lib\site-packages\sklearn\utils\validation.py", line 650, in check_array
    array = _ensure_sparse_format(array, accept_sparse=accept_sparse,
  File "d:\code\udacity\disaster-response-classifier\.venv\lib\site-packages\sklearn\utils\validation.py", line 438, in _ensure_sparse_format
    spmatrix = spmatrix.astype(dtype)
  File "d:\code\udacity\disaster-response-classifier\.venv\lib\site-packages\scipy\sparse\data.py", line 72, in astype
    self._deduped_data().astype(dtype, casting=casting, copy=copy),
  File "d:\code\udacity\disaster-response-classifier\.venv\lib\site-packages\scipy\sparse\data.py", line 32, in _deduped_data
    self.sum_d[...]<jupyter_text>7. Test your modelShow the accuracy, precision, and recall of the tuned model.  Since this project focuses on code quality, process, and  pipelines, there is no minimum performance metric needed to pass. However, make sure to fine tune your models for accuracy, precision and recall to make your project stand out - especially for your portfolio!<jupyter_code>y_pred = cv.predict(X_test)
cv_res = eval_metrics_per_category(y_test, y_pred)
cv_res

cv_res[['precision_score', 'recall_score', 'f1_score']].sort_values('f1_score').plot.bar(figsize=(20, 10))
macro_cv_res = cv_res.mean()
pd.DataFrame([macro_cv_res, macro_res]).T.plot.bar()<jupyter_output><empty_output><jupyter_text>8. Try improving your model further. Here are a few ideas:* try other machine learning algorithms* add other features besides the TF-IDF<jupyter_code>pipeline = Pipeline([
    ('union', FeatureUnion([
        ('count', Pipeline([
            ('vect', CountVectorizer(tokenizer=tokenize)),
            ('tfidf', TfidfTransformer()),
        ])),
        ('question', QuestionMarkFeature()),
        ('txtlen', TextLengthFeature())
    ])),
    ('clf', MultiOutputClassifier(RandomForestClassifier()))
])
max_samples = -1  # Used for testing ideas with subsets of the data
pipeline.fit(X_train.iloc[:max_samples], y_train.iloc[:max_samples])
y_pred = pipeline.predict(X_test)
y_pred = pd.DataFrame(y_pred, columns=y_test.columns)
print(classification_report(y_test, y_pred, target_names=y_test.columns))
pipeline.get_params()
parameters = {
    'union__count__tfidf__sublinear_tf': [True, False],
    'union__count__vect__min_df': [1, 5, 20],
    'union__count__vect__ngram_range': [(1, 1), (1, 2), (1, 3)],
    'clf__estimator__max_depth': [5, 8],
    'clf__estimator__n_estimators': [100, 200, 300],
    'clf__n_jobs': [-1],
    'clf__estimator__class_weight': ['balanced']
}

cv = GridSearchCV(pipeline, parameters, verbose=4)
cv.fit(X, Y)
pipeline.set_params(**cv.best_params_)
pipeline.fit(X_train, y_train)
y_pred = pipeline.predict(X_test)
y_pred = pd.DataFrame(y_pred, columns=y_test.columns)
print(classification_report(y_test, y_pred, target_names=y_test.columns))

res = eval_metrics_per_category(y_test, y_pred)
res[['precision_score', 'recall_score', 'f1_score', 'accuracy_score']].sort_values('f1_score').plot.bar(figsize=(20, 6))<jupyter_output>d:\code\udacity\disaster-response-classifier\.venv\lib\site-packages\sklearn\metrics\_classification.py:1248: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
d:\code\udacity\disaster-response-classifier\.venv\lib\site-packages\sklearn\metrics\_classification.py:1248: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
d:\code\udacity\disaster-response-classifier\.venv\lib\site-packages\sklearn\metrics\_classification.py:1248: UndefinedMetricWarning: Recall is ill-defined and being set to 0.0 due to no true samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
d:\code\udacity\disaster-response-classifier\.ve[...]<jupyter_text>Test NB model<jupyter_code>nb_pipeline = Pipeline(
    [
        ('vect', CountVectorizer(tokenizer=tokenize)),
        ('tfidf', TfidfTransformer()),
        ('clf', MultiOutputClassifier(MultinomialNB()))
    ]
)

nb_params = {
    'tfidf__sublinear_tf': True,
    'vect__ngram_range': (1, 3)
    }

nb_pipeline.set_params(**nb_params)
nb_pipeline.fit(X_train, y_train)
y_pred = nb_pipeline.predict(X_test)
y_pred = pd.DataFrame(y_pred, columns=y_test.columns)
print(classification_report(y_test, y_pred, target_names=y_test.columns))
nb_res = eval_metrics_per_category(y_test, y_pred)
nb_res[['precision_score', 'recall_score', 'f1_score', 'accuracy_score']].sort_values('f1_score').plot.bar(figsize=(20, 6))<jupyter_output>d:\code\udacity\disaster-response-classifier\.venv\lib\site-packages\sklearn\metrics\_classification.py:1248: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
d:\code\udacity\disaster-response-classifier\.venv\lib\site-packages\sklearn\metrics\_classification.py:1248: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
d:\code\udacity\disaster-response-classifier\.venv\lib\site-packages\sklearn\metrics\_classification.py:1248: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
d:\code\udacity\disaster-response-classi[...]<jupyter_text>Naive Bayes model does not appear to be a good fit for this task  9. Export your model as a pickle file<jupyter_code>model_filepath = 'classifier.pkl'
model = pipeline

with open(model_filepath, 'wb') as out:
        joblib.dump(model, out)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11590211661027618, 'fraction_numerical': 0.03021480840349359, 'mean_word_length': 5.193957115009747, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 37, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '151520', 'license': 'MIT', 'n_tokens_mistral': 4530, 'n_tokens_neox': 4259, 'n_words': 1020, 'path': 'notebooks/ML Pipeline Preparation.ipynb', 'repo_name': 'guischmitd/disaster-response-classifier', 'size': '12709'}