<jupyter_start><jupyter_text>Data Cleaning: Correcting, Completing, Creating, and Converting & Feature Engineering  Data Visualisation<jupyter_code>mails.groupby('label')['length'].plot.hist(bins=150)<jupyter_output><empty_output><jupyter_text>Looks like spam messages density is between 150 to 200 words length<jupyter_code>mails['length'].describe()<jupyter_output><empty_output><jupyter_text>max message length is 910. Let's see what this message says<jupyter_code>mails[mails['length']==910]['message'].iloc[0]
mails.hist(column='length',by='label',bins=60,figsize=(12,6))
spam_words = ' '.join(list(mails[mails['label']=='spam']['message']))
spam_wc = WordCloud(width=500,height=500).generate(spam_words)
plt.figure(figsize=(12,8),facecolor='K')
plt.imshow(spam_wc)
plt.axis('off')
plt.tight_layout(0)<jupyter_output><empty_output><jupyter_text>As expected spam messages mostly contains "FREE" , "CALL NOW" etc.  Noise RemovalAny piece of text which is not relevant to the context of the data and the end-output can be specified as the noise.For example – language stopwords (commonly used words of a language – is, am, the, of, in etc), URLs or links, social media entities (mentions, hashtags), punctuations and industry specific words.<jupyter_code>def text_process(mess):
    """
    1. remove punc
    2. remove stop words
    3. return list of clean text words
    
    """
    nopunc=[char for char in mess if char not in string.punctuation]
    
    nopunc = ''.join(nopunc)
    
    return [word for word in nopunc.split() if word.lower not in stopwords.words('english')]
#Here is the original data once again
mails.head()<jupyter_output><empty_output><jupyter_text>Now let's "tokenize" these messages. Tokenization is just the term used to describe the process of converting the normal text strings in to a list of tokens (words that we actually want).  VectorizationWe will go ahead and use scikit CountVectorizer to convert our bag of words into Vector form, where each of rows will represents unique words and each of columns represent our actual text messages. Here, i am using own analyzer but you can use defauit analyzer as well.<jupyter_code>bow_transformer=CountVectorizer(analyzer=text_process).fit(mails['message'])<jupyter_output><empty_output><jupyter_text>Let's check how many unique vocabulary we have into our data set<jupyter_code>print(len(bow_transformer.vocabulary_))
messages_bow=bow_transformer.transform(mails['message'])
print('Shape of Sparse Matrix: ', messages_bow.shape)
messages_bow.nnz<jupyter_output><empty_output><jupyter_text>TF - IDF  - Count how many times does a word occur in each message (Known as term frequency) - Weigh the counts, so that frequent tokens get lower weight (inverse document frequency) - Normalize the vectors to unit length, to abstract from the original text length (L2 norm)<jupyter_code>tfidf_transformer=TfidfTransformer().fit(messages_bow)
# Just checking inverse document frequency of single word
tfidf_transformer.idf_[bow_transformer.vocabulary_['Love']]<jupyter_output><empty_output><jupyter_text>** Now i will convert entire Bag of words corpus to tfidf corpus **<jupyter_code>messages_tfidf=tfidf_transformer.transform(messages_bow)<jupyter_output><empty_output><jupyter_text>Training our dataset using Naive bayes classifier<jupyter_code>from sklearn.model_selection import train_test_split<jupyter_output><empty_output><jupyter_text>Splitting my dataset into train and test<jupyter_code>msg_train,msg_test,label_train,label_test=train_test_split(mails['message'],mails['label'],test_size=0.3)
from sklearn.pipeline import Pipeline<jupyter_output><empty_output><jupyter_text>Now i will use sklearn pipeline that is very efficient and fast way to do all of the things what i have done before in this notebook manually just to show all the hidden maths and techniques of text processing.<jupyter_code># Here in few lines of code we can preprocess text's. It is scikit capablities that gives fast way to do these short of the things.
pipeline=Pipeline([
    ('bow',CountVectorizer(analyzer=text_process)),
    ('tfidf',TfidfTransformer()),
    ('classifier',MultinomialNB())
])<jupyter_output><empty_output><jupyter_text>Now i will fit my training set and predict outputs on test set.<jupyter_code>pipeline.fit(msg_train,label_train)
predictions=pipeline.predict(msg_test)
print(confusion_matrix(label_test,predictions))
print('\n')
print(classification_report(label_test,predictions))<jupyter_output>[[1464    0]
 [  75  133]]


             precision    recall  f1-score   support

        ham       0.95      1.00      0.98      1464
       spam       1.00      0.64      0.78       208

avg / total       0.96      0.96      0.95      1672
{'fraction_non_alphanumeric': 0.09377646062658763, 'fraction_numerical': 0.018204911092294666, 'mean_word_length': 5.8379160636758325, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 48, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '635527', 'license': 'Apache-2.0', 'n_tokens_mistral': 1530, 'n_tokens_neox': 1437, 'n_words': 440, 'path': 'Spam_Classification.ipynb', 'repo_name': 'akum001/data_science_portfolio', 'size': '4724'}