<jupyter_start><jupyter_text>ML Pipeline PreparationFollow the instructions below to help you create your ML pipeline. 1. Import libraries and load data from database.- Import Python libraries- Load dataset from database with [`read_sql_table`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_sql_table.html)- Define feature and target variables X and Y<jupyter_code># import libraries
import pandas as pd
from sqlalchemy import create_engine

# NLP
import re
import nltk
from nltk.corpus import stopwords
from nltk.stem.wordnet import WordNetLemmatizer
from nltk.tokenize import word_tokenize

nltk.download('punkt')
nltk.download('stopwords')
nltk.download('wordnet')

# sklearn
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.multioutput import MultiOutputClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.metrics import classification_report, f1_score
from sklearn.externals import joblib
# from sklearn.metrics import confusion_matrix
# 
# from sklearn.base import BaseEstimator, TransformerMixin

# load data from database
engine = create_engine('sqlite:///messages.db')
connection = engine.connect()
df = pd.read_sql_table("messages", con=connection)
X = df.iloc[:, 1]
Y = df.iloc[:, 4:]<jupyter_output><empty_output><jupyter_text>2. Write a tokenization function to process your text data<jupyter_code>def tokenize(text):
    # normalize case and remove punctuation
    text = re.sub(r"[^a-zA-Z0-9]", " ", text.lower())
    
    # tokenize text
    tokens = word_tokenize(text)
    
    # lemmatize andremove stop words
    stop_words = stopwords.words("english")
    lemmatizer = WordNetLemmatizer()
    tokens = [lemmatizer.lemmatize(word) for word in tokens if word not in stop_words]

    return tokens<jupyter_output><empty_output><jupyter_text>3. Build a machine learning pipelineThis machine pipeline should take in the `message` column as input and output classification results on the other 36 categories in the dataset. You may find the [MultiOutputClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.multioutput.MultiOutputClassifier.html) helpful for predicting multiple target variables.<jupyter_code>pipeline = Pipeline([
    ('vect', CountVectorizer(tokenizer=tokenize)),
    ('tfidf', TfidfTransformer()),
    ('mcfl', MultiOutputClassifier(estimator=RandomForestClassifier(n_estimators = 10)))
])<jupyter_output><empty_output><jupyter_text>4. Train pipeline- Split data into train and test sets- Train pipeline<jupyter_code>X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.3, random_state=42)

pipeline.fit(X_train, Y_train)<jupyter_output><empty_output><jupyter_text>5. Test your modelReport the f1 score, precision and recall for each output category of the dataset. You can do this by iterating through the columns and calling sklearn's `classification_report` on each.<jupyter_code>def show_metrics(Y_test, Y_pred):
    """
    Print f1-score, precision and recall for each category
    
    Parameters:
    -----------
    Y_test: True values,
    Y_pred: Predicted values
    """
    for cat in range(0, Y_test.shape[1]):
        cf=classification_report(Y_test.iloc[:, cat], pd.DataFrame(Y_pred).iloc[:, cat], output_dict=True).get('1')
        if cf is not None:
            print(f"Category:{cat: 3d}, f1-score: {cf['f1-score'] :5.3f}, precision: {cf['precision']: 5.3f}, recall: {cf['recall']: 5.3f}")
        else:
            print(f"Caterogy:{cat: 3d} doesn't have value 1")
                  
Y_pred = pipeline.predict(X_test)                  
show_metrics(Y_test, Y_pred)
print(classification_report(Y_test, Y_pred))<jupyter_output>/Users/stingl/anaconda3/envs/nd025_aktuell/lib/python3.7/site-packages/sklearn/metrics/classification.py:1145: UndefinedMetricWarning: Recall and F-score are ill-defined and being set to 0.0 in labels with no true samples.
  'recall', 'true', average, warn_for)<jupyter_text>6. Improve your modelUse grid search to find better parameters.<jupyter_code>pipeline.get_params()
parameters = {
#     'vect__binary': [True, False], 
    'mcfl__estimator__max_depth': [20, 50, 100],
    'mcfl__estimator__min_samples_split': [2, 5, 20],
    'mcfl__estimator__n_estimators': [5, 10, 20]
}

cv = GridSearchCV(pipeline, param_grid=parameters, verbose=10, cv=3, n_jobs=12, scoring='f1_micro')

cv.fit(X_train, Y_train)
cv.best_params_
cv.best_score_
cv.score(X_test, Y_test)<jupyter_output><empty_output><jupyter_text>7. Test your modelShow the accuracy, precision, and recall of the tuned model.  Since this project focuses on code quality, process, and  pipelines, there is no minimum performance metric needed to pass. However, make sure to fine tune your models for accuracy, precision and recall to make your project stand out - especially for your portfolio!<jupyter_code>Y_pred = cv.predict(X_test)

show_metrics(Y_test, Y_pred)
print(classification_report(Y_test, Y_pred))<jupyter_output>precision    recall  f1-score   support

           0       0.80      0.98      0.88      5941
           1       0.86      0.39      0.54      1333
           2       0.00      0.00      0.00        34
           3       0.78      0.63      0.70      3286
           4       0.67      0.04      0.08       644
           5       0.79      0.05      0.10       414
           6       0.75      0.01      0.02       239
           7       0.33      0.01      0.01       156
           8       0.71      0.04      0.07       267
           9       0.00      0.00      0.00         0
          10       0.90      0.20      0.33       512
          11       0.85      0.28      0.42       878
          12       0.89      0.20      0.33       714
          13       0.83      0.04      0.08       123
          14       1.00      0.01      0.02       192
          15       0.00      0.00      0.00        88
          16       0.50      0.02      0.04       259
          17       0.91    [...]<jupyter_text>8. Try improving your model further. Here are a few ideas:* try other machine learning algorithms* add other features besides the TF-IDF  Drop columns with only 1 value"child_alone" and maybe other columns have only one value. So drop them because they may cause problems with some Classificators.<jupyter_code>def drop_single_value_columns(data):
    """
    Drop any column which doesn't contain two diffent values
    
    Parameters:
    -----------
    data: DataFrame
    
    Return value:
    -------------
    DataFrame with column(s) removed
    """
    
    data.columns[data.nunique(axis=0) != 2]
    data.drop(columns=data.columns[data.nunique(axis=0) != 2], inplace=True)
    return data
X = df.iloc[:, 1]
Y = drop_single_value_columns(df.iloc[:, 4:]) 


X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.3, random_state=42)

pipeline = Pipeline([
    ('vect', CountVectorizer(tokenizer=tokenize)),
    ('tfidf', TfidfTransformer()),
    ('mcfl', MultiOutputClassifier(estimator=SVC()))
])

parameters = {
#     'mcfl__estimator__kernel' : ['linear', 'rbf', 'sigmoid'],
    'mcfl__estimator__kernel' : ['linear'],
#     'mcfl__estimator__degree' : [3, 4, 5, 6],
#     'mcfl__estimator__kernel' : ['poly'],
}

cv = GridSearchCV(pipeline, param_grid=parameters, verbose=10, cv=3, n_jobs=12, scoring='f1_micro')

cv.fit(X_train, Y_train)
Y_pred = cv.predict(X_test)
print(cv.best_params_)
print(cv.best_score_)
print(classification_report(Y_test, Y_pred)) 
X = df.iloc[:, 1]
Y = df.iloc[:, 4:].drop(columns='child_alone')

X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.3, random_state=42)

pipeline = Pipeline([
    ('vect', CountVectorizer(tokenizer=tokenize)),
    ('tfidf', TfidfTransformer()),
    ('mcfl', MultiOutputClassifier(estimator=SVC()))
])

parameters = {
#     'mcfl__estimator__kernel' : ['linear', 'rbf', 'sigmoid'],
    'mcfl__estimator__degree' : [2, 3, 4, 5, 6],
    'mcfl__estimator__kernel' : ['poly'],
}

cv = GridSearchCV(pipeline, param_grid=parameters, verbose=10, cv=3, n_jobs=12, scoring='f1_micro')

cv.fit(X_train, Y_train)
Y_pred = cv.predict(X_test)
print(cv.best_params_)
print(cv.best_score_)
print(classification_report(Y_test, Y_pred)) 
pipeline.get_params()
from sklearn.svm import LinearSVC
X = df.iloc[:, 1]
Y = df.iloc[:, 4:].drop(columns='child_alone')

X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.3, random_state=42)

pipeline = Pipeline([
    ('vect', CountVectorizer(tokenizer=tokenize)),
    ('tfidf', TfidfTransformer()),
    ('mcfl', MultiOutputClassifier(estimator=LinearSVC()))
])

parameters = {
    'mcfl__estimator__C' : [1],
    'mcfl__estimator__loss' : ['hinge', 'squared_hinge'],
    'mcfl__estimator__max_iter' : [1000, 10000]
}

cv = GridSearchCV(pipeline, param_grid=parameters, verbose=10, cv=3, n_jobs=12, scoring='f1_micro')

cv.fit(X_train, Y_train)
Y_pred = cv.predict(X_test)
print(cv.best_params_)
print(cv.best_score_)
print(classification_report(Y_test, Y_pred))
cv.get_params()
from sklearn.naive_bayes import MultinomialNB
X = df.iloc[:, 1]
Y = df.iloc[:, 4:] #.drop(columns='child_alone')

X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.3, random_state=42)

pipeline = Pipeline([
    ('vect', CountVectorizer(tokenizer=tokenize)),
    ('tfidf', TfidfTransformer()),
    ('mcfl', MultiOutputClassifier(estimator=MultinomialNB()))
])

parameters = {
    'mcfl__estimator__alpha' : [0.01, 0.1, 1, 10]
}

cv = GridSearchCV(pipeline, param_grid=parameters, verbose=10, cv=3, n_jobs=12, scoring='f1_micro')

cv.fit(X_train, Y_train)
Y_pred = cv.predict(X_test)
print(cv.best_params_)
print(cv.best_score_)
print(classification_report(Y_test, Y_pred))
cv.get_params()<jupyter_output><empty_output><jupyter_text>9. Export your model as a pickle file<jupyter_code>joblib.dump(cv.best_estimator_, 'classifier.pkl', compress = 1)
cls = joblib.load('classifier.pkl')
pd.DataFrame(cls.predict(X_test)).head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11914641375222289, 'fraction_numerical': 0.03813475597707963, 'mean_word_length': 3.6952690166975883, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '184063', 'license': 'MIT', 'n_tokens_mistral': 3679, 'n_tokens_neox': 3470, 'n_words': 823, 'path': 'jupyter-notebooks/ML Pipeline Preparation.ipynb', 'repo_name': 'dfv-ms/udacity_nd025_term2_project_2', 'size': '10122'}