<jupyter_start><jupyter_text>**Data Import & Snapshot of the data**<jupyter_code># Load data in a dataframe
dt = pd.read_csv('SPAM-210331-134237.csv')

# Snapshot of the data - 10 items
dt.head(10)<jupyter_output><empty_output><jupyter_text>**Mapping**Map the class 'spam' to 1 (int) and 'ham' to 0 (int).<jupyter_code># Normalization: Mapping 'spam' to 1 (int) and 'ham' to 0 (int)
dt['spam'] = dt['type'].map({'spam':1, 'ham':0}).astype(int)

dt.head() # Snapshot of updated dataframe<jupyter_output><empty_output><jupyter_text>List the names of columns in the updated datafram:<jupyter_code>print('Columns in the given data:')
for col in dt.columns:
    print(col)
type_len = len(dt['type'])
print('Number of rows in the review column:', type_len)

text_len = len(dt['text'])
print('Number of rows in the liked column:', text_len)<jupyter_output>Number of rows in the review column: 116
Number of rows in the liked column: 116<jupyter_text>2. Tokenization<jupyter_code>dt['text'][1] # before<jupyter_output><empty_output><jupyter_text>**Tokenization**Tokenization is changing a sentence or multiple sentences into a bag of words.<jupyter_code>def tokenizer(text):
    return text.split()
dt['text'] = dt['text'].apply(tokenizer)
dt['text'][1] # after<jupyter_output><empty_output><jupyter_text>**Stemming**Stemming is reducing all the derived words to their base words. For e.g., changing growing, grown, etc. to 'grow'.We can anyone of the three different tools:1. Snowball1. Porter1. Lancaster<jupyter_code>dt['text'][48] # before
# Stemming

from nltk.stem.snowball import SnowballStemmer
porter = SnowballStemmer('english', ignore_stopwords = False)
def stem_it(text):
    """Assumes text to be a list of strings.
       Returns --> a list of strings of the same size
       but having the words replaced by their respective
       root forms."""
    return [porter.stem(word) for word in text]
dt['text'] = dt['text'].apply(stem_it)
dt['text'][48] # after stemming<jupyter_output><empty_output><jupyter_text>**Lemmitization**<jupyter_code>dt['text'][92] # before
from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
def lemmatize_it(text):
    return [lemmatizer.lemmatize(word, pos = 'a') for word in text]
dt['text'] = dt['text'].apply(lemmatize_it)
dt['text'][92] # after<jupyter_output><empty_output><jupyter_text>**Stopword Removal**<jupyter_code>dt['text'][34] # before
from nltk.corpus import stopwords
stop_words = stopwords.words('english')
def stop_it(text):
    review = [word for word in text if not word in stop_words]
    return review
dt['text'] = dt['text'].apply(stop_it)
dt['text'][34] # after
dt['text'] = dt['text'].apply(' '.join)
dt.head(10)<jupyter_output><empty_output><jupyter_text>**Vectorization**It is the process of changing the sentence (a string) to an array of numbers. The resulting vector should incorporate the frequency and significance of a word in a document.To vectorize a document, we use here, _**Term Frequency - Inverse Document Frequency (TF-IDF)**_<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer
tfidf = TfidfVectorizer()
y = dt.spam.values
x = tfidf.fit_transform(dt['text'])
x
y
print(len(y))<jupyter_output>116<jupyter_text>**Split the data into two categories: train-test spit**<jupyter_code>from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(x, y, random_state = 1, test_size = 0.2, shuffle = False)
print("Items in train split:", len(y_train), "\nItems in test split:", len(y_test))<jupyter_output>Items in train split: 92 
Items in test split: 24<jupyter_text>**Classification using Logistic Regression**<jupyter_code>from sklearn.linear_model import LogisticRegression
clf = LogisticRegression()
clf.fit(x_train, y_train)
y_pred = clf.predict(x_test)
from sklearn.metrics import accuracy_score
acc_log = accuracy_score(y_pred, y_test)*100
print("Accuracy:", acc_log)<jupyter_output>Accuracy: 87.5<jupyter_text>**Classification using LinearSVC Accuracy**<jupyter_code>from sklearn.svm import LinearSVC

linear_svc = LinearSVC(random_state = 0)
linear_svc.fit(x_train, y_train)
y_pred = linear_svc.predict(x_test)
acc_linear_svc = accuracy_score(y_pred, y_test)*100
print("Accuracy:", acc_linear_svc)<jupyter_output>Accuracy: 87.5
{'fraction_non_alphanumeric': 0.12870132898111447, 'fraction_numerical': 0.016553975285614364, 'mean_word_length': 6.814207650273224, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '29979', 'license': 'MIT', 'n_tokens_mistral': 1564, 'n_tokens_neox': 1499, 'n_words': 388, 'path': 'email-spam-detection.ipynb', 'repo_name': 'amnksngh/Gmail-Spam-Detection', 'size': '4289'}