<jupyter_start><jupyter_text>COMP9318 Lab4  Instructions1. This note book contains instructions for **COMP9318-Lab4**.* You are required to complete your implementation in a file `submission.py` provided along with this notebook.* You are not allowed to print out unnecessary stuff. We will not consider any output printed out on the screen. All results should be returned in appropriate data structures via corresponding functions.* You can submit your implementation for **Lab4** via following link: https://kg.cse.unsw.edu.au/submit/ .* For each question, we have provided you with detailed instructions along with question headings. In case of any problem, you can post your query @ Piazza.* You are allowed to add other functions and/or import modules (you may have to in this lab), but you are not allowed to define global variables. **Only functions are allowed** in `submission.py`. * You should not import unnecessary modules/libraries, failing to import such modules at test time will lead to errors.* We will provide immediate feedback on your submission. You can access your scores using the online submission portal on the same day. * For **Final Evaluation** we will be using a different dataset, so your final scores may vary.  * You are allowed to submit as many times as you want before the deadline, but **ONLY the latest version will be kept and marked**.* Submission deadline for this assignment is **23:59:59 on 16th April, 2019**. We will **not** accept any late submissions.  Question-1: Text Classification using Multinomial Naive BayesYou are required to implement a multinomial naive bayes classifier to predict spam SMS.The training data is a set of SMS categoried into `spam` and `ham`.<jupyter_code>import pandas as pd

raw_data = pd.read_csv('./asset/data.txt', sep='\t')
raw_data.head()<jupyter_output><empty_output><jupyter_text>In order to implement a unigram model, first we tokenize the text. We use the count corresponding to each token (word) in the SMS as its feature (i.e., bag of words). We store the features and catrgorical information for each SMS in a `dictionary`.<jupyter_code>def tokenize(sms):
    return sms.split(' ')

def get_freq_of_tokens(sms):
    tokens = {}
    for token in tokenize(sms):
        if token not in tokens:
            tokens[token] = 1
        else:
            tokens[token] += 1
    return tokens

training_data = []
for index in range(len(raw_data)):
    training_data.append((get_freq_of_tokens(raw_data.iloc[index].text), raw_data.iloc[index].category))<jupyter_output><empty_output><jupyter_text>For this lab, you need to **implement** a multinomial naive bayes classifier (i.e., `multinomial_nb()` in the file: `submission.py`) with add-1 smoothing. The input arguments of `multinomial_nb()` are:* `training_data`: pre-processed data stored as a `dictionary`* `sms`: test-sms (i.e., a list of tokens) that you need to categorize as `spam` and/or `ham`The return value of `multinomial_nb()` should be the **ratio** of the probability of sms is spam and the probability of sms is ham. A return value larger than 1 implies the `sms` is spam and vice versa.For example, a sample output is shown in the cell given below:<jupyter_code>## How we test your implementation...
import submission_ans as submission

sms = 'I am not spam'
print(submission.multinomial_nb(training_data, tokenize(sms)))<jupyter_output>0.2342767295597484<jupyter_text>Test EnvironmentFor testing, we have pre-installed the requisite modules and/or libraries in the testing environment. You are only allowed to use following libraries:* python: 3.6.5* pandas: 0.19.2NOTE: You are required to implement the classifier by yourself. You are not allowed to import **sklearn** and/or any other library in Lab4.<jupyter_code>import submission as submission

sms = 'I am not spam'
print(submission.multinomial_nb(training_data, tokenize(sms)))<jupyter_output>0.23427672955974846
{'fraction_non_alphanumeric': 0.08092928261424559, 'fraction_numerical': 0.018381414347715088, 'mean_word_length': 4.856502242152466, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '500395', 'license': 'MIT', 'n_tokens_mistral': 1149, 'n_tokens_neox': 1065, 'n_words': 505, 'path': 'Lab4/Lab4-Specs.ipynb', 'repo_name': 'Bennyhwanggggg/Data-Warehousing-and-Data-Mining', 'size': '3917'}