<jupyter_start><jupyter_text>Naive Bayes classifier and Online Learning using Numpy  1. Naive Bayes classifier for MNIST* Step by step implementation of a Naive Bayes classifier that support discrete and continuous features for each handwritten digit.* Input:    1. We will use image training data from [MNIST](http://yann.lecun.com/exdb/mnist/)        * Download the MNIST from this website and parse it.        * Please read the description in the link to understand the format.        * Basically, each image is represented by bits (Whole binary file is in big endian format; we need to deal with it), we can use a char arrary to store an image.        * There are some headers we need to deal with as well, please read the link for more details.    2. Training lable data from MNIST.    3. Testing image from MNIST    4. Testing label from MNIST    5. Toggle option        0: discrete mode        1: continuous mode* Output:    1. we print out the posterior (in log scale to avoid underflow) of the ten categories (0-9) for each image input.     2. We have to marginalize posterior so that its sum is equal to 1.    3. For each test image, we print out our prediction which is the category having the highest posterior, and tally the prediction by comparing with input.    4. Then we print out the imagination of numbers in Bayes classifier.    ![image-3.png](attachment:image-3.png)    5. We calculate and report the error rate in the end.* Functionality:    1. In Discrete mode:        * Tally the frequency of the values of each pixel into 32 bins. For example, pixel 0 to 7 should be classified to bin 1, pixel 8 to 15 should be bin 2 ... etc. Then perform Naive Bayes classifier. Note that to avoid empty bin, you can use a peudocount (such as the minimum value in other bins) for instead.    2. In Continuous mode:        * Use MLE to fit a Gaussian distribution for the value of each pixel. Perform Naive Bayes classifier.* Follow chapter-2 of [**Pattern Recognition and Machine learning**](http://users.isr.ist.utl.pt/~wurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf) book to understand theory of Naive Bayes classifier from Bayesian perspective.<jupyter_code>import numpy as np
from functools import reduce
%matplotlib inline
from functools import reduce
import matplotlib.pyplot as plt
from math import log, sqrt, exp, pi, factorial, erf
class load_file():
    def __init__(self, file):
        self.filename = file
        self.dset = []
        
        with open(self.filename, mode='rb') as f:
            magic_number = f.read(4) # read first four bytes
            
            self.n_dim = magic_number[-1] #n_dim=3
            
            self.n_dims = [int.from_bytes(f.read(4), byteorder='big') for _ in range(self.n_dim)] #n_dims=[60000,28,28]
            #read a vector of size 784 for each iteration 
            for idx in range(self.n_dims[0]):
                item = f.read(reduce(lambda x,y: x*y, (self.n_dims[1:] + [1]))) 
        
                self.dset.append(item)
file_1 = './project-02/train-images.idx3-ubyte'
file_2 = './project-02/train-labels.idx1-ubyte'
file_3 = './project-02/t10k-images.idx3-ubyte'
file_4 = './project-02/t10k-labels.idx1-ubyte'
train_x = load_file(file_1)
train_y = load_file(file_2)
test_x  = load_file(file_3)
test_y  = load_file(file_4)
len(train_x.dset), len(train_y.dset), len(test_x.dset), len(test_y.dset)
def showImage(byte, row = 28, col = 28):
    im = [[byte[i*28 + j] for j in range(col)] for i in range(row)]
    
    plt.imshow(im, cmap='gray')
data_idx = 10
showImage(train_x.dset[data_idx])
print("labels [{}] : {}".format(data_idx, train_y.dset[data_idx][0]))<jupyter_output>labels [10] : 3<jupyter_text>Discrete<jupyter_code># cnt_pixels[class][pixels][bins]
def cnt_pixclass(train_x, train_y, w = 28, h = 28, nc = 10, bins = 10**-6):
    
    pixels_cnt = [[[bins for _ in range(32)] for _ in range(w * h)] for _ in range(nc)]
    
    classes_cnt = [0 for _ in range(nc)]

    for indx, data in enumerate(train_x):
        
        
        classes_cnt[train_y[indx][0]] += 1
        
        for i in range(len(data)):
            pixels_cnt[train_y[indx][0]][i][data[i] >> 3] += 1
                
    return classes_cnt, pixels_cnt
def fit_posterior_disc(test_x, test_y, classes_cnt, pixels_cnt):
    posterior = [[0 for _ in range(10)] for _ in range(len(test_x))]
    accuracy = 0
    
    for indx, data in enumerate(test_x):
        for y in range(10):
            for i in range(len(data)):
                
                posterior[indx][y] += log(pixels_cnt[y][i][data[i] >> 3])
                
            posterior[indx][y] -= (len(data) - 1 ) * log(classes_cnt[y])
            #posterior[idx][label] += log(cnt_classes[label])
            
        pred = np.argmax(posterior[indx])
        answ = test_y[indx][0]
        
        if pred == answ:
            accuracy += 1
    print("Accuracy on test images:", accuracy/len(test_y))
    return posterior, accuracy 
def predict(test_labels, posterior,indx):
    p=np.array(posterior[indx])/np.array(posterior[indx]).sum()
    print('Posterior (in log scale):\n')
    for key in (range(0,len(p))):
        print(key,':',p[key]) #print marginalized posterior 

    print("index: {}\t label : {}\t predict : {}".format(indx, test_y.dset[indx][0], np.argmax(posterior[indx])))
def imagination_discrete(count_pixels,label=0):
    pixel_sum_015=0
    pixel_sum_031=0
    img_num =[0 for _ in range (28*28)]
    for i in range (784):
        for k in range(32):
            if k<16:
                pixel_sum_015=pixel_sum_015+count_pixels[label][i][k]
            else:
                pixel_sum_031=pixel_sum_031+count_pixels[label][i][k]
            
        if pixel_sum_015>pixel_sum_031:
            img_num[i]=0
            pixel_sum_015=0
        else:
            img_num[i]=1
            pixel_sum_031=0 
    img_num=np.array(img_num).reshape(28,28)
    print(img_num)
    
    #plt.imshow(img_num)<jupyter_output><empty_output><jupyter_text>Run Discrete Naive Bayes<jupyter_code>cnt_classes, cnt_pixels = cnt_pixclass(train_x.dset, train_y.dset)
posterior, accuracy = fit_posterior_disc(test_x.dset, test_y.dset, cnt_classes, cnt_pixels)
predict(test_y, posterior,indx=6)
imagination_discrete(cnt_pixels,label=3)
"Discrete Error rate : {}".format(1-(accuracy/len(test_x.dset)))<jupyter_output><empty_output><jupyter_text>Continous<jupyter_code>def extract_pixels(train_x,train_y):
    pixels = [[[] for _ in range(28*28)] for _ in range(10)]

    for indx, data in enumerate(train_x):
        for i in range(len(data)):
            pixels[train_y[indx][0]][i].append(data[i])
            
    return pixels
    
def normal(u, v):
    return lambda x : (exp(-1*(((x - u)**2) / (2*v) ) ) / sqrt(2*pi*v))

def pdf1(u, v):
    return lambda x : 0.5*(1 + erf((x - u)/(sqrt(2*v))))

def pdf(x, w, cdf):
    a = cdf(x - w/2)
    b = cdf(x + w/2)
    return b - a
def sample_u_v(pixels):
    pixel_normal = [[[0, 0] for _ in range(28*28)] for _ in range(10)]
    var = 10**3

    for i in range(10):
        for j in range(28*28):
        
            n = len(pixels[i][j])
            u = sum(pixels[i][j]) / n
            v = sum([(x - u)**2 for x in pixels[i][j]]) / n
            pixel_normal[i][j][0] = u
        
            if v <= var:
                v = var
            pixel_normal[i][j][1] = v
            pixel_normal[i][j].append(normal(u, v))
            pixel_normal[i][j].append(pdf1(u, v))
        
    return pixel_normal
def fit_post_cont(test_x, test_y, pixel_normal, cnt_classess):
    accuracy = 0
    width = 1
    ln0 = 10**-10
    posterior = [[0 for _ in range(10)] for _ in range(len(test_x))]
    for indx, data in enumerate(test_x):
        for k in range(10):
            for i in range(28*28):
           
                p = pdf(data[i], width, pixel_normal[k][i][3])

                if p <= ln0:
                    posterior[indx][k] += log(ln0)
                    continue
                
                posterior[indx][k] += log(p)
            posterior[indx][k] += log(cnt_classes[k])
    
        pred = np.argmax(posterior[indx])
        answ = test_y[indx][0]
    
        if pred == answ:
            accuracy += 1
   
    return posterior, accuracy
def imagination_continous(pixels,label=3):
    pixel_mean=0
    img_num =[0 for _ in range (28*28)]
    for i in range (784):
        pixel_mean =pixels[label][i][0]
            
        if pixel_mean<=127:
            img_num[i]=0
            pixel_mean=0
        else:
            img_num[i]=1 
            pixel_mean=0
    img_num=np.array(img_num).reshape(28,28)
    print(img_num)<jupyter_output><empty_output><jupyter_text>Run continous Naive Bayes<jupyter_code>pixels=extract_pixels(train_x.dset,train_y.dset)
pixel_normal=sample_u_v(pixels)
poster, accuracy1 =fit_post_cont(test_x.dset, test_y.dset, pixel_normal, cnt_classes)
"Continuous accuracy: {}".format(accuracy/len(test_y.dset))
predict(test_y.dset, poster,indx=9)
imagination_continous(pixel_normal,label=9)
"Discrete Error rate : {}".format(1-(accuracy1/len(test_y.dset)))<jupyter_output><empty_output><jupyter_text>2. Online Learning * We use online learning to learn the beta distribution of the parameter p (chance to see 1) of the cointossing trails in batch.* Input:    1. A file (tosscoin.txt) contains many lines of binary outcomes:    2. Parameter a for the initial beta prior    3. Parameter b for the initial beta prior* Output: print out the Binomial likelihood (based on MLE), Beta prior and posterior probability (parameters only) for each line.* Functionality: Use Beta-Binomial conjugation to perform online learning.<jupyter_code>tosscoin=[]
filename='./project-02/tosscoin.txt'

with open(filename,'r') as f:
    tosses=f.readlines()
    for trail in tosses:
        N=len(trail[:-1])
        m=0
        for i in range(len(trail)):
            if trail[i]=='1':
                m+=1
        tosscoin.append((N,m))
print(tosscoin)
def online_learning(tosscoin, init_a = 0, init_b = 0):
    beta_a = init_a
    beta_b = init_b

    print("Prior(a,b) : ({},{})".format(beta_a, beta_b))
    for idx, (N,m) in enumerate(tosscoin):

        prior_a = beta_a
        prior_b = beta_b
        beta_a += m
        beta_b += N-m

        print("Binomial likelihood: {:.3f}\t Prior(a,b): ({},{})\t Posterior(a,b): ({},{})".format(m/N, prior_a, prior_b, beta_a, beta_b))
online_learning(tosscoin, init_a=0, init_b=0)
online_learning(tosscoin, init_a=10, init_b=1)<jupyter_output>Prior(a,b) : (10,1)
Binomial likelihood: 0.591	 Prior(a,b): (10,1)	 Posterior(a,b): (23,10)
Binomial likelihood: 0.714	 Prior(a,b): (23,10)	 Posterior(a,b): (28,12)
Binomial likelihood: 0.750	 Prior(a,b): (28,12)	 Posterior(a,b): (37,15)
Binomial likelihood: 0.902	 Prior(a,b): (37,15)	 Posterior(a,b): (74,19)
Binomial likelihood: 0.591	 Prior(a,b): (74,19)	 Posterior(a,b): (87,28)
Binomial likelihood: 0.571	 Prior(a,b): (87,28)	 Posterior(a,b): (91,31)
Binomial likelihood: 0.583	 Prior(a,b): (91,31)	 Posterior(a,b): (98,36)
Binomial likelihood: 0.902	 Prior(a,b): (98,36)	 Posterior(a,b): (209,48)
Binomial likelihood: 0.538	 Prior(a,b): (209,48)	 Posterior(a,b): (237,72)
Binomial likelihood: 0.578	 Prior(a,b): (237,72)	 Posterior(a,b): (263,91)
Binomial likelihood: 0.458	 Prior(a,b): (263,91)	 Posterior(a,b): (285,117)
Binomial likelihood: 0.512	 Prior(a,b): (285,117)	 Posterior(a,b): (307,138)
Binomial likelihood: 0.488	 Prior(a,b): (307,138)	 Posterior(a,b): (328,160)
Binomial likelih[...]
{'fraction_non_alphanumeric': 0.1319726714520453, 'fraction_numerical': 0.04099282193202456, 'mean_word_length': 2.9986168741355463, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '39956', 'license': 'Apache-2.0', 'n_tokens_mistral': 4386, 'n_tokens_neox': 4069, 'n_words': 1067, 'path': '_notebooks/2020-11-20-Naive-Bayes-classifier-for-MNIST.ipynb', 'repo_name': 'rashidch/aiblog', 'size': '11563'}