<jupyter_start><jupyter_text>CS180 Independent Study This project uses fastai to build a classifier for spicules arine microorganisms. more about fastai: https://github.com/fastai  Environment setup These three lines of code should be on the top of every noteook, to get automatic reloading and inline plotting.<jupyter_code>%reload_ext autoreload
%autoreload 2
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Import libraries.<jupyter_code>from fastai.imports import *
from fastai.transforms import *
from fastai.conv_learner import *
from fastai.model import *
from fastai.dataset import *
from fastai.sgdr import *
from fastai.plots import *<jupyter_output><empty_output><jupyter_text>Setting the path to the data. The data folder should be placed under a same folder with this notebook file.I'm going to use cross validation in this project, so I create 6 folders, in which contain different randomly selected data for tarinning/validing.<jupyter_code>#PATH = "data/cs180/"
PATH1 = "data/cs180-test1/"
PATH2 = "data/cs180-test2/"
PATH3 = "data/cs180-test3/"
PATH4 = "data/cs180-test4/"
PATH5 = "data/cs180-test5/"
PATH6 = "data/cs180-test6/"<jupyter_output><empty_output><jupyter_text>Every test folder cotains a train folder and vaild folder. There are positive cases and negative cases in both the train and vaild folder. I distribute the data as below:data/cs180-testX =>  train => positives => (76  positive cases)                              negatives => (106 negative cases)                                         vaild => positives => (18 positive cases)                              negatives => (24 negative cases) Set image size to 225 * 225. Since the original files have relatively low resolutions, 225 is more than enough.<jupyter_code>sz = 225<jupyter_output><empty_output><jupyter_text>First run configure all 6 tests<jupyter_code>arch=resnet34
data1 = ImageClassifierData.from_paths(PATH1, tfms=tfms_from_model(arch, sz))
learn1 = ConvLearner.pretrained(arch, data1, precompute=True)

arch=resnet34
data2 = ImageClassifierData.from_paths(PATH2, tfms=tfms_from_model(arch, sz))
learn2 = ConvLearner.pretrained(arch, data2, precompute=True)

arch=resnet34
data3 = ImageClassifierData.from_paths(PATH3, tfms=tfms_from_model(arch, sz))
learn3 = ConvLearner.pretrained(arch, data3, precompute=True)

arch=resnet34
data4 = ImageClassifierData.from_paths(PATH4, tfms=tfms_from_model(arch, sz))
learn4 = ConvLearner.pretrained(arch, data4, precompute=True)

arch=resnet34
data5 = ImageClassifierData.from_paths(PATH5, tfms=tfms_from_model(arch, sz))
learn5 = ConvLearner.pretrained(arch, data5, precompute=True)

arch=resnet34
data6 = ImageClassifierData.from_paths(PATH6, tfms=tfms_from_model(arch, sz))
learn6 = ConvLearner.pretrained(arch, data6, precompute=True)<jupyter_output><empty_output><jupyter_text>Run tests, since learning rate finder doesn't work well because I don't have enough data. I maunally tested different numbers for learning rate, I found out (0.08, 0.11) worked out pretty good.<jupyter_code>%time learn1.fit(0.1, 20)
%time learn2.fit(0.1, 20)
%time learn3.fit(0.1, 20)
%time learn4.fit(0.1, 20)
%time learn5.fit(0.1, 20)
%time learn6.fit(0.1, 20)<jupyter_output><empty_output><jupyter_text>Learning rate finder.<jupyter_code>#learn = learn1
#data = data1
#learn = ConvLearner.pretrained(arch, data, precompute=True)
#lrf=learn.lr_find()
#learn.sched.plot_lr()
#learn.sched.plot()<jupyter_output><empty_output><jupyter_text>Run 2, with more data generated from modifying the original data New imagine can be upside down, max zoom is 1.1.<jupyter_code>tfms = tfms_from_model(resnet34, sz, aug_tfms=transforms_top_down, max_zoom=1.1)<jupyter_output><empty_output><jupyter_text>The code below is for taking a peek at the new imagines.<jupyter_code>def get_augs():
    data = ImageClassifierData.from_paths(PATH1, bs=2, tfms=tfms, num_workers=1)
    x,_ = next(iter(data.aug_dl))
    return data.trn_ds.denorm(x)[1]
ims = np.stack([get_augs() for i in range(6)])
plots(ims, rows=2)<jupyter_output><empty_output><jupyter_text>run test<jupyter_code>data1 = ImageClassifierData.from_paths(PATH1, tfms=tfms)
learn1 = ConvLearner.pretrained(arch, data1, precompute=True)

data2 = ImageClassifierData.from_paths(PATH2, tfms=tfms)
learn2 = ConvLearner.pretrained(arch, data2, precompute=True)

data3 = ImageClassifierData.from_paths(PATH3, tfms=tfms)
learn3 = ConvLearner.pretrained(arch, data3, precompute=True)

data4 = ImageClassifierData.from_paths(PATH4, tfms=tfms)
learn4 = ConvLearner.pretrained(arch, data4, precompute=True)

data5 = ImageClassifierData.from_paths(PATH5, tfms=tfms)
learn5 = ConvLearner.pretrained(arch, data5, precompute=True)

data6 = ImageClassifierData.from_paths(PATH6, tfms=tfms)
learn6 = ConvLearner.pretrained(arch, data6, precompute=True)


%time learn1.fit(0.1, 20)

%time learn2.fit(0.1, 20)

%time learn3.fit(0.1, 20)

%time learn4.fit(0.1, 20)

%time learn5.fit(0.1, 20)

%time learn6.fit(0.1, 20)<jupyter_output><empty_output><jupyter_text>saving our models to prevent losts in case of out of GPU mem.<jupyter_code>learn1.save('1_lastlayer')
learn2.save('2_lastlayer')
learn3.save('3_lastlayer')
learn4.save('4_lastlayer')
learn5.save('5_lastlayer')
learn6.save('6_lastlayer')
learn1.load('1_lastlayer')
learn2.load('2_lastlayer')
learn3.load('3_lastlayer')
learn4.load('4_lastlayer')
learn5.load('5_lastlayer')
learn6.load('6_lastlayer')<jupyter_output><empty_output><jupyter_text>Run 3, with precompute=False<jupyter_code>learn1.precompute=False
learn2.precompute=False
learn3.precompute=False
learn4.precompute=False
learn5.precompute=False
learn6.precompute=False
print('test1')
%time learn1.fit(0.1, 2, cycle_len=3)
print('test2')
%time learn2.fit(0.1, 10, cycle_len=3)
print('test3')
%time learn3.fit(0.1, 10, cycle_len=3)
print('test4')
%time learn4.fit(0.1, 10, cycle_len=3)
print('test5')
%time learn5.fit(0.1, 10, cycle_len=3)
print('test6')
%time learn6.fit(0.1, 10, cycle_len=3)<jupyter_output>test6<jupyter_text>Run 4 with an array of learning rate. (needs 2G+ GPU memory)<jupyter_code>learn1.sched.plot_lr()
learn1.unfreeze()
lr=np.array([1e-4,1e-3,1e-2])
#%time learn1.fit(lr, 3, cycle_len=1, cycle_mult=2)
#%time learn2.fit(lr, 3, cycle_len=1, cycle_mult=2)
#%time learn3.fit(lr, 3, cycle_len=1, cycle_mult=2)
#%time learn4.fit(lr, 3, cycle_len=1, cycle_mult=2)
#%time learn5.fit(lr, 3, cycle_len=1, cycle_mult=2)
#%time learn6.fit(lr, 3, cycle_len=1, cycle_mult=2)<jupyter_output><empty_output><jupyter_text>Examine the wrong classifies We need to know which is 'negatives'/'positives'.In this case, o is negatives, 1 is positives  Test1<jupyter_code>data = data1
learn =learn1
PATH = PATH1
data.classes<jupyter_output><empty_output><jupyter_text>Below is code from fastai course, it is for looking at the result.<jupyter_code># this gives prediction for validation set. Predictions are in log scale
log_preds = learn.predict()
log_preds.shape
preds = np.argmax(log_preds, axis=1)  # from log probabilities to 0 or 1
probs = np.exp(log_preds[:,1])        # pr(dog)
#def rand_by_mask(mask): return np.random.choice(np.where(mask)[0], min(len(preds), 4), replace=False)
def rand_by_mask(mask): return np.where(mask)[0]

def rand_by_correct(is_correct): return rand_by_mask((preds == data.val_y)==is_correct)
def plots(ims, figsize=(12,6), rows=1, titles=None):
    f = plt.figure(figsize=figsize)
    for i in range(len(ims)):
        sp = f.add_subplot(rows, len(ims)//rows, i+1)
        sp.axis('Off')
        if titles is not None: sp.set_title(titles[i], fontsize=16)
        plt.imshow(ims[i])
        
def load_img_id(ds, idx): 
    return np.array(PIL.Image.open(PATH+ds.fnames[idx]))

def plot_val_with_title(idxs, title):
    imgs = [load_img_id(data.val_ds,x) for x in idxs]
    title_filenames = [data.val_ds.fnames[x] for x in idxs]
    title_probs = [probs[x] for x in idxs]
    title_filenames_and_probs = []
    for i in range(len(imgs)):
        title_filenames_and_probs.append(title_filenames[i] + "\n" + title_probs[i].astype('str'))
        #IF THERE ARE TOO MANY CASES, UNCOMMENT THE NEXT LINE TO PRINT ALL THE FILENAMES AND PROBS
        #print(title_filenames_and_probs[i])
    return plots(imgs, rows=1, titles=title_filenames_and_probs, figsize=(16,8)) if len(imgs)>0 else print('Not Found.')
    
def most_by_mask(mask, mult):
    idxs = np.where(mask)[0]
    return idxs[np.argsort(mult * probs[idxs])]
   # return idxs[np.argsort(mult * probs[idxs])[:4]]

def most_by_correct(y, is_correct): 
    mult = -1 if (y==1)==is_correct else 1
    return most_by_mask(((preds == data.val_y)==is_correct) & (data.val_y == y), mult)


plot_val_with_title(most_by_correct(1, False), "Most incorrect positives")
plot_val_with_title(most_by_correct(0, False), "Most incorrect nagetives")<jupyter_output>valid/negatives/Negative_individualRGBs_radiolarans_tests_2__00_w75_h189.jpg
0.99934745
valid/negatives/Negative_individualRGBs_diatoms__90_w227_h96.jpg
0.6829639<jupyter_text>Test2<jupyter_code>data = data2
learn =learn2
PATH = PATH2
data.classes
# this gives prediction for validation set. Predictions are in log scale
# this gives prediction for validation set. Predictions are in log scale
log_preds = learn.predict()
log_preds.shape
preds = np.argmax(log_preds, axis=1)  # from log probabilities to 0 or 1
probs = np.exp(log_preds[:,1])        # pr(dog)
#def rand_by_mask(mask): return np.random.choice(np.where(mask)[0], min(len(preds), 4), replace=False)
def rand_by_mask(mask): return np.where(mask)[0]

def rand_by_correct(is_correct): return rand_by_mask((preds == data.val_y)==is_correct)
def plots(ims, figsize=(12,6), rows=1, titles=None):
    f = plt.figure(figsize=figsize)
    for i in range(len(ims)):
        sp = f.add_subplot(rows, len(ims)//rows, i+1)
        sp.axis('Off')
        if titles is not None: sp.set_title(titles[i], fontsize=16)
        plt.imshow(ims[i])
        
def load_img_id(ds, idx): 
    return np.array(PIL.Image.open(PATH+ds.fnames[idx]))

def plot_val_with_title(idxs, title):
    imgs = [load_img_id(data.val_ds,x) for x in idxs]
    title_filenames = [data.val_ds.fnames[x] for x in idxs]
    title_probs = [probs[x] for x in idxs]
    title_filenames_and_probs = []
    for i in range(len(imgs)):
        title_filenames_and_probs.append(title_filenames[i] + "\n" + title_probs[i].astype('str'))
        #IF THERE ARE TOO MANY CASES, UNCOMMENT THE NEXT LINE TO PRINT ALL THE FILENAMES AND PROBS
        #print(title_filenames_and_probs[i])
    return plots(imgs, rows=1, titles=title_filenames_and_probs, figsize=(16,8)) if len(imgs)>0 else print('Not Found.')
    
def most_by_mask(mask, mult):
    idxs = np.where(mask)[0]
    return idxs[np.argsort(mult * probs[idxs])]
   # return idxs[np.argsort(mult * probs[idxs])[:4]]

def most_by_correct(y, is_correct): 
    mult = -1 if (y==1)==is_correct else 1
    return most_by_mask(((preds == data.val_y)==is_correct) & (data.val_y == y), mult)
plot_val_with_title(most_by_correct(1, False), "Most incorrect positives")
plot_val_with_title(most_by_correct(0, False), "Most incorrect nagetives")<jupyter_output>Most incorrect nagetives
Not Found.<jupyter_text>Test3<jupyter_code>data = data3
learn =learn3
PATH = PATH3
data.classes
# this gives prediction for validation set. Predictions are in log scale
log_preds = learn.predict()
log_preds.shape
preds = np.argmax(log_preds, axis=1)  # from log probabilities to 0 or 1
probs = np.exp(log_preds[:,1])        # pr(dog)
#def rand_by_mask(mask): return np.random.choice(np.where(mask)[0], min(len(preds), 4), replace=False)
def rand_by_mask(mask): return np.where(mask)[0]

def rand_by_correct(is_correct): return rand_by_mask((preds == data.val_y)==is_correct)
def plots(ims, figsize=(12,6), rows=1, titles=None):
    f = plt.figure(figsize=figsize)
    for i in range(len(ims)):
        sp = f.add_subplot(rows, len(ims)//rows, i+1)
        sp.axis('Off')
        if titles is not None: sp.set_title(titles[i], fontsize=16)
        plt.imshow(ims[i])
        
def load_img_id(ds, idx): 
    return np.array(PIL.Image.open(PATH+ds.fnames[idx]))

def plot_val_with_title(idxs, title):
    imgs = [load_img_id(data.val_ds,x) for x in idxs]
    title_filenames = [data.val_ds.fnames[x] for x in idxs]
    title_probs = [probs[x] for x in idxs]
    title_filenames_and_probs = []
    for i in range(len(imgs)):
        title_filenames_and_probs.append(title_filenames[i] + "\n" + title_probs[i].astype('str'))
        #IF THERE ARE TOO MANY CASES, UNCOMMENT THE NEXT LINE TO PRINT ALL THE FILENAMES AND PROBS
        #print(title_filenames_and_probs[i])
    return plots(imgs, rows=1, titles=title_filenames_and_probs, figsize=(16,8)) if len(imgs)>0 else print('Not Found.')
    
def most_by_mask(mask, mult):
    idxs = np.where(mask)[0]
    return idxs[np.argsort(mult * probs[idxs])]
   # return idxs[np.argsort(mult * probs[idxs])[:4]]

def most_by_correct(y, is_correct): 
    mult = -1 if (y==1)==is_correct else 1
    return most_by_mask(((preds == data.val_y)==is_correct) & (data.val_y == y), mult)
plot_val_with_title(most_by_correct(1, False), "Most incorrect positives")
plot_val_with_title(most_by_correct(0, False), "Most incorrect nagetives")<jupyter_output>Most incorrect nagetives<jupyter_text>Test4<jupyter_code>data = data4
learn =learn4
PATH = PATH4
data.classes
# this gives prediction for validation set. Predictions are in log scale
log_preds = learn.predict()
log_preds.shape
preds = np.argmax(log_preds, axis=1)  # from log probabilities to 0 or 1
probs = np.exp(log_preds[:,1])        # pr(dog)
#def rand_by_mask(mask): return np.random.choice(np.where(mask)[0], min(len(preds), 4), replace=False)
def rand_by_mask(mask): return np.where(mask)[0]

def rand_by_correct(is_correct): return rand_by_mask((preds == data.val_y)==is_correct)
def plots(ims, figsize=(12,6), rows=1, titles=None):
    f = plt.figure(figsize=figsize)
    for i in range(len(ims)):
        sp = f.add_subplot(rows, len(ims)//rows, i+1)
        sp.axis('Off')
        if titles is not None: sp.set_title(titles[i], fontsize=16)
        plt.imshow(ims[i])
        
def load_img_id(ds, idx): 
    return np.array(PIL.Image.open(PATH+ds.fnames[idx]))

def plot_val_with_title(idxs, title):
    imgs = [load_img_id(data.val_ds,x) for x in idxs]
    title_filenames = [data.val_ds.fnames[x] for x in idxs]
    title_probs = [probs[x] for x in idxs]
    title_filenames_and_probs = []
    for i in range(len(imgs)):
        title_filenames_and_probs.append(title_filenames[i] + "\n" + title_probs[i].astype('str'))
        #IF THERE ARE TOO MANY CASES, UNCOMMENT THE NEXT LINE TO PRINT ALL THE FILENAMES AND PROBS
        #print(title_filenames_and_probs[i])
    return plots(imgs, rows=1, titles=title_filenames_and_probs, figsize=(16,8)) if len(imgs)>0 else print('Not Found.')
    
def most_by_mask(mask, mult):
    idxs = np.where(mask)[0]
    return idxs[np.argsort(mult * probs[idxs])]
   # return idxs[np.argsort(mult * probs[idxs])[:4]]

def most_by_correct(y, is_correct): 
    mult = -1 if (y==1)==is_correct else 1
    return most_by_mask(((preds == data.val_y)==is_correct) & (data.val_y == y), mult)
plot_val_with_title(most_by_correct(1, False), "Most incorrect positives")
plot_val_with_title(most_by_correct(0, False), "Most incorrect nagetives")<jupyter_output>Most incorrect nagetives
Not Found.<jupyter_text>Test5<jupyter_code>data = data5
learn =learn5
PATH = PATH5
data.classes
# this gives prediction for validation set. Predictions are in log scale
log_preds = learn.predict()
log_preds.shape
preds = np.argmax(log_preds, axis=1)  # from log probabilities to 0 or 1
probs = np.exp(log_preds[:,1])        # pr(dog)
#def rand_by_mask(mask): return np.random.choice(np.where(mask)[0], min(len(preds), 4), replace=False)
def rand_by_mask(mask): return np.where(mask)[0]

def rand_by_correct(is_correct): return rand_by_mask((preds == data.val_y)==is_correct)
def plots(ims, figsize=(12,6), rows=1, titles=None):
    f = plt.figure(figsize=figsize)
    for i in range(len(ims)):
        sp = f.add_subplot(rows, len(ims)//rows, i+1)
        sp.axis('Off')
        if titles is not None: sp.set_title(titles[i], fontsize=16)
        plt.imshow(ims[i])
        
def load_img_id(ds, idx): 
    return np.array(PIL.Image.open(PATH+ds.fnames[idx]))

def plot_val_with_title(idxs, title):
    imgs = [load_img_id(data.val_ds,x) for x in idxs]
    title_filenames = [data.val_ds.fnames[x] for x in idxs]
    title_probs = [probs[x] for x in idxs]
    title_filenames_and_probs = []
    for i in range(len(imgs)):
        title_filenames_and_probs.append(title_filenames[i] + "\n" + title_probs[i].astype('str'))
        #IF THERE ARE TOO MANY CASES, UNCOMMENT THE NEXT LINE TO PRINT ALL THE FILENAMES AND PROBS
        #print(title_filenames_and_probs[i])
    return plots(imgs, rows=1, titles=title_filenames_and_probs, figsize=(16,8)) if len(imgs)>0 else print('Not Found.')
    
def most_by_mask(mask, mult):
    idxs = np.where(mask)[0]
    return idxs[np.argsort(mult * probs[idxs])]
   # return idxs[np.argsort(mult * probs[idxs])[:4]]

def most_by_correct(y, is_correct): 
    mult = -1 if (y==1)==is_correct else 1
    return most_by_mask(((preds == data.val_y)==is_correct) & (data.val_y == y), mult)
plot_val_with_title(most_by_correct(1, False), "Most incorrect positives")
plot_val_with_title(most_by_correct(0, False), "Most incorrect nagetives")<jupyter_output>Most incorrect nagetives
Not Found.<jupyter_text>Test6<jupyter_code>data = data6
learn =learn6
PATH = PATH6
data.classes
# this gives prediction for validation set. Predictions are in log scale
log_preds = learn.predict()
log_preds.shape
preds = np.argmax(log_preds, axis=1)  # from log probabilities to 0 or 1
probs = np.exp(log_preds[:,1])        # pr(dog)
#def rand_by_mask(mask): return np.random.choice(np.where(mask)[0], min(len(preds), 4), replace=False)
def rand_by_mask(mask): return np.where(mask)[0]

def rand_by_correct(is_correct): return rand_by_mask((preds == data.val_y)==is_correct)
def plots(ims, figsize=(12,6), rows=1, titles=None):
    f = plt.figure(figsize=figsize)
    for i in range(len(ims)):
        sp = f.add_subplot(rows, len(ims)//rows, i+1)
        sp.axis('Off')
        if titles is not None: sp.set_title(titles[i], fontsize=16)
        plt.imshow(ims[i])
        
def load_img_id(ds, idx): 
    return np.array(PIL.Image.open(PATH+ds.fnames[idx]))

def plot_val_with_title(idxs, title):
    imgs = [load_img_id(data.val_ds,x) for x in idxs]
    title_filenames = [data.val_ds.fnames[x] for x in idxs]
    title_probs = [probs[x] for x in idxs]
    title_filenames_and_probs = []
    for i in range(len(imgs)):
        title_filenames_and_probs.append(title_filenames[i] + "\n" + title_probs[i].astype('str'))
        #IF THERE ARE TOO MANY CASES, UNCOMMENT THE NEXT LINE TO PRINT ALL THE FILENAMES AND PROBS
        #print(title_filenames_and_probs[i])
    return plots(imgs, rows=1, titles=title_filenames_and_probs, figsize=(16,8)) if len(imgs)>0 else print('Not Found.')
    
def most_by_mask(mask, mult):
    idxs = np.where(mask)[0]
    return idxs[np.argsort(mult * probs[idxs])]
   # return idxs[np.argsort(mult * probs[idxs])[:4]]

def most_by_correct(y, is_correct): 
    mult = -1 if (y==1)==is_correct else 1
    return most_by_mask(((preds == data.val_y)==is_correct) & (data.val_y == y), mult)

plot_val_with_title(most_by_correct(0, False), "Most incorrect nagetives")
plot_val_with_title(most_by_correct(1, False), "Most incorrect positives")<jupyter_output>Most incorrect positives
Not Found.
{'fraction_non_alphanumeric': 0.1312630880024516, 'fraction_numerical': 0.02676336891567496, 'mean_word_length': 5.182507104515314, 'pattern_counts': {'":': 0, '<': 74, '<?xml version=': 0, '>': 87, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '291524', 'license': 'Apache-2.0', 'n_tokens_mistral': 7676, 'n_tokens_neox': 7403, 'n_words': 1613, 'path': 'cs180-1-Copy1.ipynb', 'repo_name': 'Codyyu36/marine-microorganism-images-ML', 'size': '19579'}