<jupyter_start><jupyter_text>Preparing I save experiment data (data on task spesification google classroom) into Github<jupyter_code>#get data from source
!git clone https://github.com/fajarmuslim/dataset.git
#data processing
import pandas as pd

#grid search cv to get best parameter
from sklearn.model_selection import GridSearchCV

#ML model that used on this experiment
from sklearn.naive_bayes import MultinomialNB
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier

#metrics to evaluate model
from sklearn.metrics import accuracy_score, f1_score, recall_score, precision_score

import string
from nltk.stem import SnowballStemmer

#library for natural language processing
import nltk
nltk.download('stopwords')
from nltk.corpus import stopwords

#tf idf
from sklearn.feature_extraction.text import TfidfVectorizer
#read dataset
train_data = pd.read_csv("dataset/spam/training_data.csv")
test_data = pd.read_csv("dataset/spam/testing_data.csv")
val_data = pd.read_csv("dataset/spam/val_data.csv")<jupyter_output><empty_output><jupyter_text>Exploratory Dana Analysis  See data<jupyter_code>train_data.head()
val_data.head()
test_data.head()<jupyter_output><empty_output><jupyter_text>Data size<jupyter_code>print("ukuran training data")
print(train_data.shape)
print("ukuran val data")
print(val_data.shape)
print("ukuran testing data")
print(test_data.shape)
len_total_data = len(train_data) + len(test_data) + len(val_data)
print("data total ada ", len_total_data, " baris")
print("presentase training data : ", (len(train_data) / len_total_data)*100, "%")
print("presentase validation data : ", (len(val_data) / len_total_data)*100, "%")
print("presentase testing data : ", (len(test_data) / len_total_data)*100, "%")<jupyter_output>presentase testing data :  10.00179888469149 %<jupyter_text>Ukuran training sudah cukup besar jika dibandingkan dengan ukuran testing dan validation. Hal ini menunjukkan data sudah cukup bagus  See whether the data contain null value or not<jupyter_code>print("nilai null pada train data")
print(train_data.isnull().sum())
print("nilai null pada val data")
print(val_data.isnull().sum())
print("nilai null pada test data")
print(test_data.isnull().sum())<jupyter_output>nilai null pada test data
type    0
text    0
dtype: int64<jupyter_text>Pada data tidak terdapat nilai null sehingga bisa dilanjutkan ke proses berikutnya  Preprocessing<jupyter_code>import sys 
!{sys.executable} -m pip install pyspellchecker<jupyter_output>Requirement already satisfied: pyspellchecker in /usr/local/lib/python3.6/dist-packages (0.5.5)<jupyter_text>Remove punctuation Text data contain punctuation that doesn't adding value on this spam classificationPunctuation can be act as noiseSo, i decide to remove puctuation<jupyter_code>def remove_puctuation(text):
  return text.translate(str.maketrans('','', string.punctuation))<jupyter_output><empty_output><jupyter_text>Lowering case Text contain upper case and lowercase charSo, i decide to lowering case for all word in the data<jupyter_code>def lowering_case(text):
  return text.lower()<jupyter_output><empty_output><jupyter_text>Remove Stopword Stopword doen't add value to spam or not, because stopword show on both of text spam or text not spam frequently and have netral meaningSo, i decide to remove stopword<jupyter_code>def remove_stopwords(text):
  return [word for word in text.split() if word.lower() not in stopwords.words('english')]<jupyter_output><empty_output><jupyter_text>Doing stemming Reducing inflected (or sometimes derived) words to their word stemBy doing stemming, many word that have close semantics will have close characterSo, i decide to use stemming to preprocess text<jupyter_code>def doing_stemming(array_of_words):
  words = ""
  for i in array_of_words:
    stemmer = SnowballStemmer("english")
    words += (stemmer.stem(i))+" "
  return words<jupyter_output><empty_output><jupyter_text>Apply all preprocess on the data<jupyter_code>text_train = train_data['text'].copy()
text_train = text_train.apply(remove_puctuation)
text_train = text_train.apply(lowering_case)
text_train = text_train.apply(remove_stopwords)
text_train = text_train.apply(doing_stemming)

text_val = val_data['text'].copy()
text_val = text_val.apply(remove_puctuation)
text_val = text_val.apply(lowering_case)
text_val = text_val.apply(remove_stopwords)
text_val = text_val.apply(doing_stemming)

text_test = test_data['text'].copy()
text_test = text_test.apply(remove_puctuation)
text_test = text_test.apply(lowering_case)
text_test = text_test.apply(remove_stopwords)
text_test = text_test.apply(doing_stemming)<jupyter_output><empty_output><jupyter_text>Feature Extraction TF-IDF to extract feature TF IDF shows how important a word to a document in a coppusIn this experiment i use TF IDF to extract feature from text. The reason is TF IDF can intent the important of word<jupyter_code>def extract_features(text_train, text_val, text_test):

  vectorizer = TfidfVectorizer("english")
  
  #fit and tranform text into vector
  features_train = vectorizer.fit_transform(text_train)

  #transform text using vectorizer on text_train
  features_val = vectorizer.transform(text_val)

  #transform text using vectorizer on text_train
  features_test = vectorizer.transform(text_test)
  
  return features_train, features_val, features_test
#extract features
features_train, features_val, features_test = extract_features(text_train, text_val, text_test)
#convert string label into int label
def is_spam(spam_or_ham):
  if(spam_or_ham == "spam"):
    return 1
  elif(spam_or_ham == "ham"):
    return 0
#extract label
labels_train = train_data['type'].apply(is_spam)
labels_test = test_data['type'].apply(is_spam)
labels_val = val_data['type'].apply(is_spam)<jupyter_output><empty_output><jupyter_text>Classification  Training and Validation In this stage i use some of machine learning model1.   MultinomialNB (Probabilistic based)2.   SVC (Optimum separating line/plane)3.   RandomForestClassifier (Tree based)The reason for the choice is because I will try to use different kinds of machine learning with different learning types. So, i will have good benchmarking For every machine learning model that used here, the step is : 1.   Doing parameter tuning to get best parameter2.   Validate model using validation data  Hyperparameter Tuning MultinomialNB<jupyter_code>def hyperparameter_tuning_multinomialNB(features_train, labels_train):
  parameter_candidates = [
    {'alpha': [0.01, 0.05, 0.1, 0.3 ,0.5, 1.0, 3.0]},
  ]

  # Create a GridSearchCV object with the classifier model and parameter candidates
  clf = GridSearchCV(estimator=MultinomialNB(), param_grid=parameter_candidates, n_jobs=-1)

  # Train the classifier on train feature and labels train
  clf.fit(features_train, labels_train) 

  # Print accuracy score
  print('Best score for data1:', clf.best_score_)

  # Print best parameters for the model found using grid search
  print('Best alpha:',clf.best_estimator_.alpha) 
hyperparameter_tuning_multinomialNB(features_train, labels_train)
# Validate the model using best parameters found by the grid search
MultinomialNB(alpha=0.1).fit(features_train, labels_train).score(features_val, labels_val)<jupyter_output><empty_output><jupyter_text>Hyperparameter Tuning SVC<jupyter_code>def hyperparameter_tuning_svc(features_train, labels_train):
  parameter_candidates = [
    {'C': [1, 10, 100, 1000], 'gamma': [0.1, 0.001, 0.0001], 'kernel': ['rbf', 'sigmoid']},
  ]

  # Create a GridSearchCV object with the classifier model and parameter candidates
  grid_search_cv = GridSearchCV(estimator=SVC(), param_grid=parameter_candidates, n_jobs=-1)

  # Train the classifier on train feature and labels train
  grid_search_cv.fit(features_train, labels_train) 

  # Print accuracy score
  print('Best score for data1:', grid_search_cv.best_score_)

  # Print best parameters for the model found using grid search
  print('Best C:',grid_search_cv.best_estimator_.C) 
  print('Best Kernel:',grid_search_cv.best_estimator_.kernel)
  print('Best Gamma:',grid_search_cv.best_estimator_.gamma)
hyperparameter_tuning_svc(features_train, labels_train)
# Validate the model using best parameters found by the grid search
SVC(C=1000, kernel='rbf', gamma=0.001).fit(features_train, labels_train).score(features_val, labels_val)<jupyter_output><empty_output><jupyter_text>Hyperparameter Tuning RandomForestClassifier<jupyter_code>def hyperparameter_tuning_RandomForestClassifier(features_train, labels_train):
  parameter_candidates = [
    {'n_estimators':[50, 100, 300],'criterion':['gini', 'entropy']},
  ]

  # Create a GridSearchCV object with the classifier model and parameter candidates
  clf = GridSearchCV(estimator=RandomForestClassifier(), param_grid=parameter_candidates, n_jobs=-1)

  # Train the classifier on train feature and labels train
  clf.fit(features_train, labels_train) 

  # Print accuracy score
  print('Best score for data1:', clf.best_score_)

  # Print best parameters for the model found using grid search
  print('Best n_estimators:',clf.best_estimator_.n_estimators) 
  print('Best criterion:',clf.best_estimator_.criterion) 
# Validate the model using best parameters found by the grid search
hyperparameter_tuning_RandomForestClassifier(features_train, labels_train)
RandomForestClassifier(n_estimators=300, criterion='gini').fit(features_train, labels_train).score(features_val, labels_val)<jupyter_output><empty_output><jupyter_text>Testing Stage<jupyter_code>#this function used to test using testing_data.csv
def testing(features_train, features_test, labels_train, labels_test):
    
    #list of model used on validation phase
    models = [
      ['MultinomialNB: ', MultinomialNB(alpha=0.1)],
      ['SVC: ', SVC(C=1000, kernel='rbf', gamma=0.001)],
      ['RandomForestClassifier', RandomForestClassifier(n_estimators=300, criterion='gini')]
    ]

  
    model_data = []
    for name,current_model in models :
      #current model will save metric score for current model
      current_model_data = {}
      current_model_data["model name"] = name

      #traning phase
      current_model.fit(features_train, labels_train)
      
      #testing phase
      #predict on testing data to know how better model predict on unseen data and identify is it overfitting or not
      prediction_test = current_model.predict(features_test)
      
      #calculate accuracy score
      current_model_data["test_accuracy"] = accuracy_score(labels_test,prediction_test)

      #calculate f1 score
      current_model_data["test_f1"] = f1_score(labels_test,prediction_test)
      
      #calculate precision score
      current_model_data["test_precision"] = precision_score(labels_test,prediction_test)
      
      #calculate recall score
      current_model_data["test_recall"] = recall_score(labels_test,prediction_test)

      model_data.append(current_model_data)

    #convert model_data into dataframe
    return pd.DataFrame(model_data)
testing(features_train, features_test, labels_train, labels_test)<jupyter_output><empty_output><jupyter_text>This result shows that MultinomialNB achieve best result on accuracy, f1, and recall. Also still competitive on precision. So, i decide to choose MultinomialNB as a classifier model  Conclusion In this experiment, we conclude that for spam/not spam classification tasks with a given dataset (used in this experiment). **MultinomialNB** achieves the best result when prediction. This decision is based on **accuracy, precision, recall, and f1** score on testing stage given machine learning models with **preprocessing** stage by doing remove punctuation, lowering, case, remove stopword, and stemming. Then using TF-IDF to **extract features** to be fed into the machine learning model.<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08603059241105383, 'fraction_numerical': 0.010479168427279641, 'mean_word_length': 6.273509526736325, 'pattern_counts': {'":': 0, '<': 62, '<?xml version=': 0, '>': 62, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '585257', 'license': 'MIT', 'n_tokens_mistral': 3850, 'n_tokens_neox': 3655, 'n_words': 1010, 'path': '13517149_02.ipynb', 'repo_name': 'fajarmuslim/spam-classification', 'size': '11833'}