<jupyter_start><jupyter_text>ResourcesText Analytics with SklearnNaive Bayes and text classificationStanford NLPAwk 101Awk tutorial for regex  Data Collection<jupyter_code># function to import and structure data in a dataframe
# we trim spaces to be able to split the word in ngrams later 
# and convert letters to lowercase
def raw_data_to_dataframe(path):
    text = open(path).read().split("\n")
    intermediate_form = [text[i].split("\t") for i in range(len(text))]
    cheese_disease_df = DataFrame()
    cheese_disease_df["name"] = [(intermediate_form[i][1].replace(" ","")).lower() for i in range(len(intermediate_form)-1)]
    cheese_disease_df["class"] = [intermediate_form[i][0] for i in range(len(intermediate_form)-1)]
    return cheese_disease_df
# creating training and testing DataFrame
# class: 1<=>cheeseName and 2<=>diseaseName
cheese_disease_df_train = raw_data_to_dataframe("cheeseDisease.train")
cheese_disease_df_test = raw_data_to_dataframe("cheeseDisease.test")
cheese_disease_df_train.head()<jupyter_output><empty_output><jupyter_text>Bag of words model In order to train a Machine Leanring model, we need to turn the text content into numerical feature vectors.  Constructing our vocabularyWe first, assign a fixed integer id to every ngram occurring in any word of the training set. To implement the count vector by scratch, we first created a tokenizer to split every word in ngrams, e.g.  "backpain"--tokenizer--> [ack, ain, bac, ckp, kpa, pai] for trigrams.<jupyter_code># implement a trigram tokenizer
def tokenizer(word,ngram):
    ngrams = []
    global_i = 0
    j = ngram
    while global_i < (len(word) + 1 - ngram):
        i = global_i
        while j < len(word) + 1:
            ngrams.append(word[i:j])
            i = j - 1
            j += ngram - 1
        global_i +=1
        j = ngram +  global_i
    return list(np.unique(ngrams))
# tokenizing the names elements of the dataframe 
# to return list of ngrams in our cells
# Apply tokenization to the entire column names
def tokenize_names(cheese_disease_df):
    pack = []
    tokenized_names = [tokenizer(cheese_disease_df.ix[i,0], 3) for i in range(len(cheese_disease_df))]
    cheese_disease_df_tokenized = cheese_disease_df
    cheese_disease_df_tokenized["name"] = tokenized_names
    pack.append(cheese_disease_df_tokenized)
    pack.append(tokenized_names)
    return pack

result_train =  tokenize_names(cheese_disease_df_train)
cheese_disease_df_train_tokenized = result_train[0]
tokenized_train_names = result_train[1]
cheese_disease_df_test_tokenized = tokenize_names(cheese_disease_df_test)[0]
cheese_disease_df_train_tokenized.head()<jupyter_output><empty_output><jupyter_text>After tokenizing, the process consists in creating a vocabulary, i.e. a set of unique ngrams. Each ngram will be identified by an unique integer identifier.<jupyter_code>bag_of_trigrams = []
for i in range(len(tokenized_train_names)):
    bag_of_trigrams += tokenized_train_names[i]
unique_trigrams = np.unique(bag_of_trigrams)

# create a dictionary out of the previous array 
# with the following (key, value) pairs: trigram => unique_id
dict_unique_trigrams = dict()
for i, value in enumerate(unique_trigrams):
    dict_unique_trigrams.update({value:i})
n_features = len(dict_unique_trigrams)<jupyter_output><empty_output><jupyter_text>Counting occurences of featuresFor each name, we count the number of occurrences of each ngram and store them in a matrix X[i, j] with i, the number of names and j, the unique identifier for each ngram in our dictionary. Note that our features are the unique ngrams.Here's a simple method to return occurences of each ngram in our corpus of words: occurences_in_corpus = Counter(bag_of_trigrams).<jupyter_code>def create_count_vectorizer(cheese_disease_df_tokenized):
    # here we compute the occurences of trigrams for each words, 
    # i.e. each cell of the column "names"
    occurences_per_words = []
    for i in range(len(cheese_disease_df_tokenized )):
        occurences_per_words.append(dict(Counter(cheese_disease_df_tokenized .ix[i,0])))

    # creating an initial matrix of size i*j (with i = number of words)
    # and j = vector of unique ids of the trigrams in our vocabulary
    # As a result, this matrix will be filled with a lot of 0s.
    # Add one last column on order to store the rows class. 
    count_vectorizer = np.zeros(shape=(len(cheese_disease_df_tokenized),len(dict_unique_trigrams)+1))
    for i in range(len(cheese_disease_df_tokenized)):
        for j in range(len(dict_unique_trigrams)):
            for d in occurences_per_words[i].keys(): 
                if dict_unique_trigrams.get(d) == j: 
                    count_vectorizer[i,j] = occurences_per_words[i].get(d)
    
    return count_vectorizer
# creating count_vectorizer for train dataframe
count_vectorizer_train = create_count_vectorizer(cheese_disease_df_train_tokenized)
count_vectorizer_train<jupyter_output><empty_output><jupyter_text>We have computed our matrix which counts the occurences for each ngram in each word, note that the matrix is filled with a lot of 0 values, we call this a sparse matrix.   Maximum-Likelihood Estimates Under naive assumption, we have 2 assumptions: 1 - Samples are independent and identically distributed. The prior probabilities can be obtained via the maximum-likelihood estimate (i.e., the frequencies of how often each class label is represented in the training dataset:$$ P_{cheese} = \frac{numberOfCheeseNames}{overallNumberOfSamples} $$$$P_{disease} = \frac{numberOfDiseaseNames}{overallNumberOfSamples} $$2 - Features are mutually independent. Hence, the class-conditional probabilities can be calculated as a product of the individual conditional probabilities$$ P(features|class) = \prod_{i=1}^{n_{features}} P(features_i | class) $$<jupyter_code>def prior_probabilities(cheese_disease_df):
    # we first compute prior probabilities of a ngram belonging to class cheese or disease
    number_of_samples = len(cheese_disease_df["class"])
    number_of_cheese_names = sum(cheese_disease_df["class"]=='1')
    number_of_disease_names = sum(cheese_disease_df["class"]=='2')
    probability_cheese_name = number_of_cheese_names / number_of_samples
    probability_disease_name = number_of_disease_names / number_of_samples 
    prior_probabilities = [probability_cheese_name , probability_disease_name]
    return prior_probabilities
# assigning prior probabilities values of cheese and disease
cheese_prior_proba = prior_probabilities(cheese_disease_df_train)[0]
disease_prior_proba = prior_probabilities(cheese_disease_df_train)[1]<jupyter_output><empty_output><jupyter_text>Multinomial Bayes model with Laplace smoothing ($\epsilon=1$) Using MLE and the previous assumptions we can compute the class-conditional probabilities. Moreover, given the sparsity of our matrix and to avoid to compute log(0), we apply a smoothing process with a Laplace smoothing constant of value 1, to give a small weight to features with a value 0 in our count_vectorize, this yields to the following formula: $$ P(x_i|class_j) = \frac{count(x_i|y=class_j) + \epsilon}{{\sum_{i=1}^{n_{features}} (count(x_i|y=class_j) + n_{features} * \epsilon}} $$We reason in terms of our previously count_vectorizer matrix, e.g. to get the probability of feature 'bac' given that it belongs to class cheese we compute the frequency of observing 'bac' over all the samples by summing the counts of the rows of our matrix for column j = bac_id. And we divide the previous by the overall counts of all features for a given class, this is done by grouping by class and summing the counts for each column and then adding them to 2 distinct variables corresponding to our respective classes.<jupyter_code>def compute_probability_feature_given_class(cheese_disease_df,count_vectorizer,n_features):
    number_of_cheese_names = sum(cheese_disease_df["class"]=='1')
    number_of_disease_names = sum(cheese_disease_df["class"]=='2')
    count_vectorizer
    # we add a column with class {1,2} to our array and sort it 
    # in order to be able to return the sum of all ngrams belonging to the respective classes
    # and the total occurences of every ngrams for each class. 
    for i in range(len(cheese_disease_df)):
        if cheese_disease_df.ix[i,1] == '1':
            count_vectorizer[i, n_features]  = 1
        elif  cheese_disease_df.ix[i,1] == '2':
            count_vectorizer[i, n_features]  = 2
    count_vectorizer_sorted = count_vectorizer.view(np.ndarray)
    count_vectorizer_sorted = count_vectorizer_sorted[np.lexsort((count_vectorizer[:,n_features], ))]
    frequency_of_each_trigram_perclass =  np.zeros(shape=(2, n_features ))
    
    # counting occurences of trigrams per class and total trigrams per class
    for j in range(n_features):
        frequency_of_each_trigram_perclass[0,j] = (count_vectorizer_sorted[0:number_of_cheese_names-1].sum(axis=0))[j]
        frequency_of_each_trigram_perclass[1,j] = (count_vectorizer_sorted[number_of_cheese_names:number_of_disease_names].sum(axis=0))[j]
    total_count_of_all_features_in_class_cheese = (frequency_of_each_trigram_perclass).sum(axis=1)[0]
    total_count_of_all_features_in_class_disease = (frequency_of_each_trigram_perclass).sum(axis=1)[1]
    
    # computing the conditional probabilities and filling their values in a (2, n_features) dimension matrix
    epsilon = 1
    probability_features_given_class = np.zeros(shape=(2, n_features ))
    for j in range(n_features):
        probability_features_given_class[0, j] = (frequency_of_each_trigram_perclass[0, j] + epsilon) / (total_count_of_all_features_in_class_cheese + n_features * epsilon) 
        probability_features_given_class[1, j] = (frequency_of_each_trigram_perclass[1, j] + epsilon) / (total_count_of_all_features_in_class_disease + n_features * epsilon) 
    
    return probability_features_given_class
# we compute the conditional probabilities for the training set using the previous function and return 
# a (2, n_features) dimension matrix
conditional_probabilities_train = compute_probability_feature_given_class(cheese_disease_df_train,count_vectorizer_train,n_features)<jupyter_output><empty_output><jupyter_text>Decision Rule  The best class in NB classification is the most likely or maximum a posteriori ( MAP ) class $$c_{map} = argmax(P(class | features))= argmax(P(class)\prod_{i=1}^{n_{features}} P(features_i | class)) $$However, the product of the probabilities result in a floating point underflow, so we compute it by adding the logs of the probabilities instead of multiplying the probabilities. Hence: $$c_{map} = argmax(log(P(class))+ \sum_{i=1}^{n_{features}} log(P(features_i | class))) $$ To predict whether a name from a test set belong to a given class, we use the following decision rule: IF P(cheese|featuresFromTest) > P(disease|featuresFromTest) => classify as cheese ELSE classify as disease<jupyter_code>def return_most_likely_class(cheese_disease_df, count_vectorizer,conditional_probabilities, cheese_prior_proba, disease_prior_proba, dict_unique_trigrams):
    # we initialize the weights of the class with the log(prior_probabilities) values 
    # we compute other necessary variables to get the posteriot probabilities
    c_cheese = np.log10(cheese_prior_proba)
    c_disease = np.log10(disease_prior_proba)
    number_of_cheese_names = sum(cheese_disease_df_train["class"]=='1')
    number_of_disease_names = sum(cheese_disease_df_train["class"]=='2')
    count_vectorizer_sorted = count_vectorizer.view(np.ndarray)
    count_vectorizer_sorted = count_vectorizer_sorted[np.lexsort((count_vectorizer[:,n_features], ))]
    total_occurences_features_cheese = sum(count_vectorizer_sorted[0:number_of_cheese_names,:-1].sum(axis=1))
    total_occurences_features_disease = sum(count_vectorizer_sorted[number_of_cheese_names:,:-1].sum(axis=1))
    predictions = []
    ids = []
    for i in range(len(cheese_disease_df)):
        for j in cheese_disease_df.ix[i, 0]:
            if j in (dict_unique_trigrams.keys()):
                # extract unique_id for every trigram for every row in the names column
                ids.append(dict_unique_trigrams.get(j))
            else: 
                c_cheese += np.log10(np.abs(1 / (total_occurences_features_cheese + n_features ) ))
                c_disease += np.log10(np.abs(1 / (total_occurences_features_disease + n_features ) ))
            
        # compute the posterior probabilities P(class | features) using logs 
        for ido in ids:       
            c_cheese += np.log10(np.abs(conditional_probabilities[0,ido]))
            c_disease += np.log10(np.abs(conditional_probabilities[1,ido]))
        # decision result which result in decision 1 or 2 given the outcome, 
        # which we then append to a predictions list which is returned
        if (c_cheese >= c_disease):
            predictions.append(1)
        else: 
            predictions.append(2)
        ids = []
        c_cheese = np.log10(cheese_prior_proba)
        c_disease = np.log10(disease_prior_proba)
    return predictions<jupyter_output><empty_output><jupyter_text>Here we predict the class of each word of the testing set:<jupyter_code>prediction_testset = return_most_likely_class(cheese_disease_df_test, count_vectorizer_train,conditional_probabilities_train, cheese_prior_proba, disease_prior_proba, dict_unique_trigrams)<jupyter_output><empty_output><jupyter_text>Assessing accuracy of our model  In order, to determine which ngram range returned the highest accuracy for our predictions, we added a ngram parameter in our tokenizer. Morevover, intuitively and by manually scanning the columns of the train file with awk, it appeared that some 3-grams are more likely to happen than 4-grams, returning a count of lines matching 'pain' and 'ain':  Hence, we showed below the precision and accuracy report by setting the ngram parameter to 3, to prove the above statements we ran the calculations of ngrams = 2, 3 and 4 by changing the value of ngram in tokenizer(cheese_disease_df.ix[i,0], ngram) at In [1026] and got the following precisions:     ngram_value    precision             2        83.67 %                     3        88.68 %                  4        86.73 %<jupyter_code># precision displaying function 
def precision_is(predictions,actual):
    count = 0 
    for i in range(len(actual)):
        if predictions[i] == int(actual.ix[i,1]):
            count += 1
    precision = count / len(actual)
    print("Accuracy: %.2f %%" % (precision * 100))
# we display the result of our manually implemented precision function or using the accuracy_score in sklearn
y_actual = list(cheese_disease_df_test.ix[:,1].astype('int'))
precision_is(prediction_testset,cheese_disease_df_test)
print("Accuracy in sklearn: %.2f %%" % ((metrics.accuracy_score(y_actual,prediction_testset))*100))
# we display precision, recall and f-score using classification_report function in sklearn.metrics
pred_pol_str = [str(i) for i in prediction_testset]
predicted_class2 = np.array(pred_pol_str)
print(metrics.classification_report(cheese_disease_df_test["class"], predicted_class2,target_names=["cheese","disease"]))<jupyter_output>precision    recall  f1-score   support

     cheese       0.81      0.88      0.85        68
    disease       0.93      0.89      0.91       128

avg / total       0.89      0.89      0.89       196
{'fraction_non_alphanumeric': 0.08148148148148149, 'fraction_numerical': 0.011500974658869395, 'mean_word_length': 4.727949385932266, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '357925', 'license': 'MIT', 'n_tokens_mistral': 5032, 'n_tokens_neox': 4706, 'n_words': 1363, 'path': 'Text Categorization.ipynb', 'repo_name': 'kevllino/cheese-or-disease', 'size': '15390'}