<jupyter_start><jupyter_text>Please note: I built and trained the machine learning model in jupyter notebook, afterwards I saved the trained model and scaler, which was used in the api. At the end of this workbook I created a class which I copied over to the clickbait.py file which I use to make predictions on the api file based on inputs from the user.   Importing packages and libraries<jupyter_code>import numpy as np 
import pandas as pd
import nltk
from nltk.corpus import stopwords
from nltk.probability import FreqDist
from nltk.stem import WordNetLemmatizer
import string as s
import re
import os
import joblib

import matplotlib.pyplot as plt
import seaborn as sns

from sklearn.naive_bayes import MultinomialNB
from sklearn.model_selection import train_test_split<jupyter_output><empty_output><jupyter_text>Load the dataset<jupyter_code>cb_data= pd.read_csv('clickbait_data.csv')
cb_data.head()
sns.countplot(x = cb_data.clickbait)<jupyter_output><empty_output><jupyter_text>Splitting the data into Train and Test sets<jupyter_code>x=cb_data.headline
y=cb_data.clickbait
train_x,test_x,train_y,test_y=train_test_split(x,y,test_size=0.25,random_state=22,stratify=cb_data['clickbait'])<jupyter_output><empty_output><jupyter_text>Analyzing the Train and Test Data<jupyter_code>print("No. of elements in training set")
print(train_x.size)
print("No. of elements in testing set")
print(test_x.size)
print(train_x.head())
print(train_y.head())
print(test_x.head())
print(test_y.head())<jupyter_output>31858                 Indian Ocean tsunami, one year after
20688    UN carries out first review of US human rights...
19125    Pirate Bay case: Internet group attacks websit...
13523             How Many Iconic '70s Films Have You Seen
3473        How Much Do You Really Know About "Doctor Who"
Name: headline, dtype: object
31858    0
20688    0
19125    0
13523    1
3473     1
Name: clickbait, dtype: int64<jupyter_text>Tokenization of Data The data is tokenized i.e. split into tokens which are the smallest or minimal meaningful units. The data is split into words.<jupyter_code>def tokenization(text):
    lst=text.split()
    return lst
train_x=train_x.apply(tokenization)
test_x=test_x.apply(tokenization)<jupyter_output><empty_output><jupyter_text>Converting to lowercase The data is converted into lowercase to avoid ambiguity between same words in different cases like 'NLP', 'nlp' or 'Nlp'.<jupyter_code>def lowercasing(lst):
    new_lst=[]
    for i in lst:
        i=i.lower()
        new_lst.append(i)
    return new_lst
train_x=train_x.apply(lowercasing)
test_x=test_x.apply(lowercasing)<jupyter_output><empty_output><jupyter_text>Removing Stopwords<jupyter_code>def remove_stopwords(lst):
    stop=stopwords.words('english')
    new_lst=[]
    for i in lst:
        if i not in stop:
            new_lst.append(i)
    return new_lst

train_x=train_x.apply(remove_stopwords)
test_x=test_x.apply(remove_stopwords)<jupyter_output><empty_output><jupyter_text>Removing punctuation The punctuations are removed to increase the efficiency of the model. They are irrelevant because they provide no added information.<jupyter_code>def remove_punctuations(lst):
    new_lst=[]
    for i in lst:
        for j in s.punctuation:
            i=i.replace(j,'')
        new_lst.append(i)
    return new_lst
train_x=train_x.apply(remove_punctuations)
test_x=test_x.apply(remove_punctuations)<jupyter_output><empty_output><jupyter_text>Removing Numbers<jupyter_code>def remove_numbers(lst):
    nodig_lst=[]
    new_lst=[]
    for i in lst:
        for j in s.digits:    
            i=i.replace(j,'')
        nodig_lst.append(i)
    for i in nodig_lst:
        if i!='':
            new_lst.append(i)
    return new_lst
train_x=train_x.apply(remove_numbers)
test_x=test_x.apply(remove_numbers)<jupyter_output><empty_output><jupyter_text>Removing extra spaces<jupyter_code>def remove_spaces(lst):
    new_lst=[]
    for i in lst:
        i=i.strip()
        new_lst.append(i)
    return new_lst
train_x=train_x.apply(remove_spaces)
test_x=test_x.apply(remove_spaces)<jupyter_output><empty_output><jupyter_text>Analyzing data after preprocessing After preprocessing the data i.e. after removing punctuation, stopwords, spaces and numbers.<jupyter_code>train_x.head()
test_x.head()<jupyter_output><empty_output><jupyter_text>Lemmatization Lemmatization in linguistics is the process of grouping together the inflected forms of a word so they can be analysed as a single item, identified by the word's lemma, or dictionary form. It involves the morphological analysis of words.In lemmatization we find the root word or base form of the word rather than just clipping some characters from the end e.g. is, are, am are all converted to its base form be in LemmatizationHere lemmatization is done using NLTK library.<jupyter_code>lemmatizer=WordNetLemmatizer()
def lemmatzation(lst):
    new_lst=[]
    for i in lst:
        i=lemmatizer.lemmatize(i)
        new_lst.append(i)
    return new_lst
# import nltk
# import ssl

# try:
#     _create_unverified_https_context = ssl._create_unverified_context
# except AttributeError:
#     pass
# else:
#     ssl._create_default_https_context = _create_unverified_https_context

# nltk.download()
train_x=train_x.apply(lemmatzation)
test_x=test_x.apply(lemmatzation)
train_x=train_x.apply(lambda x: ''.join(i+' ' for i in x))
test_x=test_x.apply(lambda x: ''.join(i+' ' for i in x))<jupyter_output><empty_output><jupyter_text>Countvectoriser This method is used to convert the text into features.<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer
cov=CountVectorizer(analyzer='word', ngram_range=(1,2),max_features=22500)
train_1=cov.fit_transform(train_x)
test_1=cov.transform(test_x)
train_arr=train_1.toarray()
test_arr=test_1.toarray()
pd.DataFrame(test_arr[:100], columns=cov.get_feature_names())<jupyter_output><empty_output><jupyter_text>Define Naive Bayes Classifier and training<jupyter_code>NB_MN=MultinomialNB()
NB_MN.fit(train_arr,train_y)
pred=NB_MN.predict(test_arr)<jupyter_output><empty_output><jupyter_text>Evaluation of Result<jupyter_code>from sklearn.metrics import f1_score,accuracy_score
print("F1 score of the model")
print(f1_score(test_y,pred))
print("Accuracy of the model")
print(accuracy_score(test_y,pred))
print("Accuracy of the model in percentage")
print(accuracy_score(test_y,pred)*100,"%")
from sklearn.metrics import confusion_matrix
print("Confusion Matrix")
print(confusion_matrix(test_y,pred))

from sklearn.metrics import classification_report
print("Classification Report")
print(classification_report(test_y,pred))
sns.set(font_scale=1.5)
cof=confusion_matrix(test_y, pred)
cof=pd.DataFrame(cof, index=[i for i in range(2)], columns=[i for i in range(2)])
plt.figure(figsize=(8,8))

sns.heatmap(cof, cmap="PuRd",linewidths=1, annot=True,square=True,cbar=False,fmt='d',xticklabels=['Non-clickbait','Clickbait'],yticklabels=['Non-clickbait','Clickbait'])
plt.xlabel("Predicted Class")
plt.ylabel("Actual Class")

plt.title("Confusion Matrix for Clickbait Classification")<jupyter_output><empty_output><jupyter_text>SAVING THE MODEL AND SCALER OBJECT FOR USE IN DEPLOYMENT<jupyter_code>joblib.dump(cov,"scaler.pkl")
joblib.dump(NB_MN,"model.pkl")<jupyter_output><empty_output><jupyter_text>RE-USABLE PREDICTION CLASS FOR DEPLOYMENT<jupyter_code>class ClickBaitModel:
    def __init__(self, text):
        
        '''
        Argument required is a JSON object as retrieved from the POST request
        JSON object = {"title":"string"}
        '''
        
        self.text = text["title"]
        
        
    def tokenization(self):
        lst=self.text.split()
        return lst
    
    
    def lowercasing(self,arg):
        new_lst=[]
        for i in arg:
            i=i.lower()
            new_lst.append(i)
        return new_lst

    
    def remove_stopwords(self, arg):
        stop=stopwords.words('english')
        new_lst=[]
        for i in arg:
            if i not in stop:
                new_lst.append(i)
        return new_lst

    
    def remove_punctuations(self, arg):
        new_lst=[]
        for i in arg:
            for j in s.punctuation:
                i=i.replace(j,'')
            new_lst.append(i)
        return new_lst

    
    def remove_numbers(self, arg):
        nodig_lst=[]
        new_lst=[]
        for i in arg:
            for j in s.digits:    
                i=i.replace(j,'')
            nodig_lst.append(i)
        for i in nodig_lst:
            if i!='':
                new_lst.append(i)
        return new_lst
    
    
    def remove_spaces(self, arg):
        new_lst=[]
        for i in arg:
            i=i.strip()
            new_lst.append(i)
        return new_lst


    
    def lemmatzation(self, arg):
        lemmatizer=nltk.stem.WordNetLemmatizer()
        new_lst=[]
        for i in arg:
            i=lemmatizer.lemmatize(i)
            new_lst.append(i)
        return new_lst
    
    
    def preprocess(self):
        temp = self.tokenization()
        temp = self.lowercasing(temp)
        temp = self.remove_stopwords(temp)
        temp = self.remove_punctuations(temp)
        temp = self.remove_numbers(temp)
        temp = self.remove_spaces(temp) 
        lemmatizer=WordNetLemmatizer()
        temp = self.lemmatzation(temp)
        temp = ''.join(i+' ' for i in temp)
        return temp  
    
    
    def predict(self,scaler,model):
        '''
        Arguments required are the scaler and model objects.
        This function will preprocess the JSON object, and return a prediction as a string.
        '''
        temp = self.preprocess()
        temp = scaler.transform([temp])
        temp_arr=temp.toarray()
        pred=model.predict(temp_arr)[0]
        return "Clickbait!" if pred == 1 else "Not Clickbait!"<jupyter_output><empty_output><jupyter_text>CODE FOR DEPLOYMENT<jupyter_code>import numpy as np 
import nltk
from nltk.corpus import stopwords
from nltk.probability import FreqDist
from nltk.stem import WordNetLemmatizer
import string as s
import re
import os
import joblib


loaded_model = joblib.load("###PATH###/model.pkl")
loaded_scaler = joblib.load("###PATH###/scaler.pkl")


title_data = ClickBaitModel(###ENTERPOSTREQUEST.BODY)
title_data.predict(loaded_scaler, loaded_model)<jupyter_output><empty_output><jupyter_text>SINGLE PREDICTION TEST BEFORE MOVING CODE TO DEPLOYMENT<jupyter_code>cb_data.head()
dummy = ClickBaitModel({"title": "Which TV Female Friend Group Do You Belong In"})
dummy.predict(loaded_scaler,loaded_model)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09588135273002078, 'fraction_numerical': 0.008596259210277726, 'mean_word_length': 4.015158692562767, 'pattern_counts': {'":': 2, '<': 76, '<?xml version=': 0, '>': 76, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '587037', 'license': 'MIT', 'n_tokens_mistral': 3689, 'n_tokens_neox': 3555, 'n_words': 849, 'path': 'machineLearning/clickbait.ipynb', 'repo_name': 'bartventer/clickbait-predictor-app', 'size': '10586'}