<jupyter_start><jupyter_text>02 - Kernel methods and SVMs** APSTA - Ecole Centrale Nantes **** <NAME>, <NAME> ** PARTICIPANTS: **(Fill in your names)**<jupyter_code>import os
import numpy as np
import matplotlib.pyplot as plt
import skimage
from skimage import io
import random


from skimage.color import rgb2gray
from skimage.transform import resize

from sklearn.svm import SVC
from sklearn.utils import shuffle<jupyter_output><empty_output><jupyter_text>1. Image classification on Caltech 101 **a)** Download images fromhttp://www.vision.caltech.edu/feifeili/Datasets.htmand run the code bellow to check the files and store the name of the classes in the list ```labelNamesAll```(Just run)<jupyter_code>## VERIFY LOCATION AND STORE LABEL NAMES

IMDIR = '101_ObjectCategories/'
labelNamesAll = []

for root, dirnames, filenames in os.walk(IMDIR):
    labelNamesAll.append(dirnames)
    #uncomment to check what is found in this folder
    #for filename in filenames:
        #f = os.path.join(root, filename)
        #if f.endswith(('.png', '.jpg', '.jpeg','.JPG', '.tif', '.gif')):
        #    print(f)

labelNamesAll = labelNamesAll[0]

#The list of all labels/directories is
print(labelNamesAll)<jupyter_output><empty_output><jupyter_text>**b. Build a reduced dataset for accelerating process.** To do so: - Consider only up to $K$ randomly drawn categories (start with a binary case)- Read only up to $N$ images for each class- Resize the images to $(imWidth*imHeight)$The dataset should consist of a - Input matrix $\mathbf{X}$ of size $(K\cdot N)\times (imWidth\cdot imHeight)$ with one image in every row of the matrix. - Output vector $\mathbf{y}$ of size $(K\cdot N)\times 1$ with the label index of each input point in $\bf X$.- the reduced list of the label names of size $K$ to map between the indices and the names.**Note than different classes may have different number of images so that the actual number of $\bf X$ and $\bf y$ is less than $K*N$**(Run and understand)<jupyter_code>#build DATASET from K categories and (up to) N images from category
K = 2
N = 40
imWidth = 100
imHeight = 100

#selection of label indices
X = np.zeros([K*N,imHeight*imWidth]) #data matrix, one image per row
#Y = np.zeros([K*N,1]) #label indices
Y = -np.ones([K*N,1]) #label indices
labelNames = []

#random.seed(a=42)


globalCount = 0
for i in range(K): 
    while True:
        lab = random.randint(0,len(labelNamesAll)-1)
        if lab not in labelNames:
            break
    #folders are named after the class label
    filedir = os.path.join(IMDIR,labelNamesAll[lab])
    print(filedir)

    #save the name of the class
    labelNames.append(labelNamesAll[lab])       

    classCount = 0
    for filename in os.listdir(filedir):
        f = os.path.join(filedir, filename)
        if f.endswith(('.jpg')) and (classCount < N):
            image = skimage.io.imread(f, as_grey=True)
            #image = skimage.io.imread(f, as_gray=True)
            image = skimage.transform.resize(image, [imHeight,imWidth],mode='constant')#,anti_aliasing=True)
            X[globalCount,:] = image.flatten()
            Y[globalCount,:] = i
            globalCount += 1
            classCount += 1

#Remove the unused entries of X and Y
print(globalCount)
X = X[:globalCount,:]
Y = Y[:globalCount,:]

#Check the stored classes
print(labelNames)
print(X.shape)
print(Y.T)<jupyter_output><empty_output><jupyter_text>**c. Split the dataset into train (80% of samples) and test (20% samples). **(Fill in)<jupyter_code>print(X_train.shape)
print(X_test.shape)
print(Y_train.T)
print(Y_test.T)<jupyter_output><empty_output><jupyter_text>**d) Create an SVC model, train it on the train set, and test it on the test set**. (Fill in and answer the questions)**Question** SVMs are intrinsically binary classifiers, can you train the SVC for K>2? How is that achieved?<jupyter_code># Create, train and test an svm model     


print("True classes",Y_test.T)
print("Predictions",Y_pred)
errors = np.sum((Y_test.ravel()!=Y_pred))
print('There were ', errors, 'errors')<jupyter_output><empty_output><jupyter_text>**e) Fill in the function bellow to computing different evaluation measures and give a performance report**Look at the formulas and definitions in https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)Start by computing the confusion matrix values TP, TN, FP, FN![image](confusion-matrix.png)**Question:** What are the different manners to compute and interpret the scores for multiple classes $K>2$ ? Implement them**Hint** Add a numerical zero eps to the denominators to prevent dividing by zero<jupyter_code># Functions to compute the errors between prediction and ground truth 

def compute_measures(Y_gt,Y_pred, positiveClass=1): #Y_gt = ground truth
    measures = dict()
    Y_len = len(Y_gt)
    
    eps = 1e-12
    
    # True positives TP
    TP = 
        
    # True negatives TN
    TN = 
    
    # False positives FP
    FP = 
        
    # False negatives FN
    FN = 

    print('TP ', TP, 'TN ', TN, 'FP', FP, 'FN', FN, 'Total', TP+TN+FP+FN)
    measures['TP'] = TP
    measures['TN'] = TN
    measures['FP'] = FP
    measures['FN'] = FN
    
    
    # Accuracy
    measures['accuracy'] = 
    
    # Precision
    measures['precision'] = 
        
    # Specificity
    measures['specificity']=
    
    # Recall
    measures['recall'] = 
    
    # F-measure
    measures['f1'] = 
    
    # Negative Predictive Value
    measures['npv'] = 
    
    # False Predictive Value
    measures['fpr'] = 
    
    print('Accuracy ', measures['accuracy'], '\n',
          'Precision', measures['precision'], '\n',
          'Recall', measures['recall'], '\n',
          'Specificity ', measures['specificity'], '\n',
          'F-measure', measures['f1'], '\n',
          'NPV', measures['npv'],'\n',
          'FPV', measures['fpr'],'\n')


    
    return measures

def micro_average(measuresList):
    microAverage = dict()
    eps = 1e-12
    
    
    
    # Accuracy
    microAverage['accuracy'] = 
    
    # Precision
    microAverage['precision'] = 
        
    # Specificity
    microAverage['specificity'] = 
    
    # Recall
    microAverage['recall'] = 
    
    # F-measure
    microAverage['f1'] = 
    
    # Negative Predictive Value
    microAverage['npv'] = 
    
    # False Predictive Value
    microAverage['fpr'] = 
    
        
    print('Accuracy ', microAverage['accuracy'], '\n',
          'Precision', microAverage['precision'], '\n',
          'Recall', microAverage['recall'], '\n',
          'Specificity ', microAverage['specificity'], '\n',
          'F-measure', microAverage['f1'], '\n',
          'NPV', microAverage['npv'],'\n',
          'FPV', microAverage['fpr'],'\n')
    
    return microAverage

def macro_average(measuresList):
    macroAverage = dict()

    # Accuracy
    macroAverage['accuracy'] = 
    
    # Precision
    macroAverage['precision'] = 
        
    # Specificity
    macroAverage['specificity']= 
    
    # Recall
    macroAverage['recall'] = 
    
    # F-measure
    macroAverage['f1'] = 
    
    # Negative Predictive Value
    macroAverage['npv'] = 
    
    # False Predictive Value
    macroAverage['fpr'] = 
    
    print('Accuracy ', macroAverage['accuracy'], '\n',
          'Precision', macroAverage['precision'], '\n',
          'Recall', macroAverage['recall'], '\n',
          'Specificity ', macroAverage['specificity'], '\n',
          'F-measure', macroAverage['f1'], '\n',
          'NPV', macroAverage['npv'],'\n',
          'FPV', macroAverage['fpr'],'\n')
    
    return macroAverage<jupyter_output><empty_output><jupyter_text>** e)** Measure the performance of the SVC model for multiple classes $K>2$**<jupyter_code>#Fill in a list of measure dictionaries taking as input a different positive class

multiclass = []
for k in range(K):
    print('For class',labelNames[k])
    multiclass.append(compute_measures(Y_test.ravel(),Y_pred, positiveClass=k))

print('Macro-average')
macro_average(multiclass)
    
print('Micro-average')
micro_average(multiclass)


from sklearn.metrics import classification_report #confusion_matrix, accuracy_score, precision_score, recall_score, f1_micro, f1_macro
print(classification_report(Y_test.ravel(), Y_pred, target_names=labelNames))<jupyter_output><empty_output><jupyter_text>**f) Show the test images as well as the the predictions (Y_pred) vs the ground truth (Y_gt) labels for the best model**(Just run for each analysed model)<jupyter_code># Show some results
width=20
height=15
plt.rcParams['figure.figsize'] = [width, height]
fig=plt.figure()
imCounter = 1
for i in range(len(Y_test)):
    image=np.reshape(X_test[i,:], (imHeight,imWidth)) 

    plt.subplot(5,7,imCounter)
    plt.imshow(image,cmap='gray')
    plt.axis('off')
    gtLabel = labelNames[Y_test.ravel()[i].astype(int)]
    predLabel = labelNames[Y_pred.ravel()[i].astype(int)]
    plt.title('GT: {}. \n Pred: {}'.format(gtLabel, predLabel))

    imCounter += 1
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13132543461410698, 'fraction_numerical': 0.00675451223563282, 'mean_word_length': 3.4757185332011895, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 38, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '165142', 'license': 'MIT', 'n_tokens_mistral': 3042, 'n_tokens_neox': 2808, 'n_words': 800, 'path': '02-TP-kernels and svm/02-LEARN-svm-with-caltech101-sv.ipynb', 'repo_name': 'mathieulagrange/master-learn', 'size': '9046'}