<jupyter_start><jupyter_text>Data exploration Problem statementThis is a classification task. The objective is given an image that may contain an ID, inference on the image to classify if (a) FULLY_VISIBILITY, (b) PARTIAL_VISIBILITY, (c) NO_VISIBILITY.* A data set of very noisy color images (corrupted sensor) for each class, however a csv file with correct annotations is provided.* No description is given for the runtime - will the user provide clear images and not noisy ones?* We will need to make the most out of this situtation and produce a solution with the existing data.   Read the csv file First we will take a look on the csv file with the annotations. * We will make some simple sanity checks - find out if csv file is corrupted/has missing values* We will investigate if the dataset is balanced or not (same number of instances per class cateogry).<jupyter_code>import pandas as pd
base_image_dir = '../data/'
csv_dir = base_image_dir+'gicsd_labels.csv'
annot_data = pd.read_csv(csv_dir, sep=',\s+', skipinitialspace=True)
annot_data.head(10)<jupyter_output><empty_output><jupyter_text>* Printing some general stats, looks already that there is a great class imbalance between the FULL_VISIBILITY and the other two classes<jupyter_code>annot_data.describe()<jupyter_output><empty_output><jupyter_text>* Checking for missing values in columns<jupyter_code># Check the missing values
dct_feat_nansnum = {}
for column in annot_data.columns:
    n_nans=annot_data[column].isnull().sum()
    dct_feat_nansnum[column] = n_nans 
    print(f'{column} NaNs: {n_nans}/{len(annot_data)}')<jupyter_output>IMAGE_FILENAME NaNs: 0/800
LABEL NaNs: 0/800<jupyter_text>* Extracting the classes names and the number of examples per class in the provided dataset<jupyter_code>l_unq_classes = annot_data['LABEL'].unique()
print(l_unq_classes)
classes_counts = annot_data['LABEL'].value_counts()
print(classes_counts)<jupyter_output>FULL_VISIBILITY       646
PARTIAL_VISIBILITY    123
NO_VISIBILITY          31
Name: LABEL, dtype: int64<jupyter_text>Tackling imbalanced datasetsAs the dataset is imbalanced several issues will arise should we choose a machine learning/deep learning approach. * In Bibliography, F1 (for binary) or FBeta (for multiclass) scores account for class imbalance when scoring the goodness of a classifier. Picking such a score function for evaluation could negate the need to manipulate with the number of example images per class.* Undersampling, oversampling and generating synthetic data is another way to tackle an imbalance set. This could negate the hastle of picking complex scoring functions for evaluation, or even more niche techniques where the objective function in training accounts for the class imbalance (rather than using e.g. accuracy as objective function during training).* Cost based classification, as well as class-reweight are some of those techniques for more complex objective functions during training. In particula to class based classification, this can be applied even after a model has been trained by reweighting the obtained class probabilities in runtime to adjust the classifier with the costs errors and class imbalance.* A good first approach for the task can be oversampling, where we replicate some points from the minority classes in order to increase its cardinality and match the majority class  Images inspection, denoising and other image preprocessing* In this bit of data exploration, we will sample a few images to get a sense of how noisy data is. * Based on the findings we will figure out a descent image-preprocessing/denoising scheme, then following the instructions we will investigate how data looks after transforming the images to single channel (grayscale). * The cleaner the image data in a computer vision solution the better the chances of success.* A more rigid and diligent methodology would be to try and compute a noise distribution that would model noise, by using color statistics from the 800 data images. Then that noise distribution would be used in image signal processing techniques.* This could be hard to model though as it appears that the images are drawn from (a) different video streams (50 videos), (b) of different sensors/phones (MIDV-500 original dataset used two phones) (c) the sensors were moving, (d) the scene was changing (hand could appear placing id)<jupyter_code>import numpy as np
import cv2
from matplotlib import pyplot as plt
import glob<jupyter_output><empty_output><jupyter_text>Non-Local Means DenoisingA denoising technique that takes more time compared to blurring techniques however it is documented that its result is very good. We give a shot to this more advanced method as:* the images are very noisy* the (a-d) facts desribed in the intro of image inspection point to a more advanced denoise method, preferablywith no consecutive frames assumption* Also as the image is too noisy, it is legid to try and denoise on the original RGB image before converting to single channel (grayscale) as instructed in the case study instruction. This in the hopes of preserving more of the original scene information. If we would convert to grayscale first and as the image is too noisy, we risk to accumulate more of the noise/escalate the information loss.* You can read more on NLmean in this link: http://www.ipol.im/pub/art/2011/bcm_nlm/<jupyter_code>dataset_path = '../data/images/'
img_name = 'GICSD_1_4_133.png' #'GICSD_2_2_73.png'#'GICSD_1_4_133.png' #'GICSD_9_5_153.png'
img = cv2.imread(dataset_path+img_name)
print(f'Denoising: {dataset_path+img_name}')
dst_light = cv2.fastNlMeansDenoisingColored(img,None,13,13,7,21)
dst_heavy = cv2.fastNlMeansDenoisingColored(img,None,21,21,7,21)
plt.subplot(131),plt.imshow(img)
plt.title('Original Image')
plt.subplot(132),plt.imshow(dst_light)
plt.title('NLmean light')
plt.subplot(133),plt.imshow(dst_heavy)
plt.title('NLmean heavy')
plt.show()<jupyter_output><empty_output><jupyter_text>* Convert NLmean_heavy to single channel - gray; this type of images can be the training data<jupyter_code>gray_dst_heavy = cv2.cvtColor(dst_heavy,cv2.COLOR_RGB2GRAY)  
plt.subplot(121),plt.imshow(gray_dst_heavy, cmap='gray')
plt.title('NLmean heavy gray')
plt.show()
print(type(gray_dst_heavy))
print(gray_dst_heavy.shape)<jupyter_output><class 'numpy.ndarray'>
(192, 192)<jupyter_text>* We notice that the more we increase the h, hColor components the more we remove noise however the more we also remoce details of the image.* This does not look to provide much better result from a simple median filter operation for our given task.  Non-Local Means Denoising using a sequence of frames from the same source that are close in time* We will use NLMean again this time on a sequence of frames from the same source to see if we will get better results* We have notice that the provided dataset corresponds to 50 videos, from which 16 images have been extracted from each one of the videos* Thus for each video, we have a list of 16 noisy frames. We specify a temporal window size which specifies the number of nearby frames (before and after - should be odd) to be used for denoising, while the challenged frame to be denoised will be in the center of the list. (We could use a qring data structure moving that temporal window and denoise every frame per video case, but for simplicity we will just inspect things manually)<jupyter_code>list_imgs = glob.glob(dataset_path+'GICSD_1_*.png')
list_imgs.sort()
idx_denoise=7
print(f'Denoising: {list_imgs[idx_denoise]}')
l_img=[]
l_img = [cv2.imread(img_path) for img_path in list_imgs]
list_imgs
dst_stream_light = cv2.fastNlMeansDenoisingColoredMulti(l_img, idx_denoise, 7, None,13,13,7,21)
dst_stream_heavy = cv2.fastNlMeansDenoisingColoredMulti(l_img, idx_denoise, 7, None,21,21,7,21)
plt.subplot(131),plt.imshow(l_img[idx_denoise])
plt.title('Original Image')
plt.subplot(132),plt.imshow(dst_stream_light)
plt.title('NLmean Seq light')
plt.subplot(133),plt.imshow(dst_stream_heavy)
plt.title('NLmean Seq heavy')
plt.show()<jupyter_output><empty_output><jupyter_text>* Convert NLmean_sq_heavy to single channel - gray; this type of images can be the training data<jupyter_code>gray_dst_stream_heavy = cv2.cvtColor(dst_stream_heavy,cv2.COLOR_RGB2GRAY)  
plt.subplot(121),plt.imshow(gray_dst_stream_heavy, cmap='gray')
plt.title('NLmean Seq heavy gray')
plt.show()<jupyter_output><empty_output><jupyter_text>* We notice some denoising gains and as the dataset is engineered to be noisy we could have applied this scheme under the complete 800 images with a qring mechanism and create a cleaner dataset.* As this can breach some of the (hidden) assumptions of the problem we decide not to use this scheme* Also as this also looks to produce similar results with a median filter - we still had to increase the h, hColor components to clear noise at the cost of detail, we decide to go ahead an try the old-faithfull traditional median filter  Median Filter Color* We will deploy the traditional median filter for (salt and pepper) noise reduction * To clean the high noise, we expect a similar loss in detail to NLmean     * first on the color image and then on the original color image, then convert the result to grayscale    * second by converting to the grayscale image and the applying median filter on it    * this way we may have a sense if our original assumption on denoising in the color is indeed better<jupyter_code>dataset_path = '../data/images/'
img_name = 'GICSD_1_4_133.png' #'GICSD_2_2_73.png'#'GICSD_1_4_133.png' #'GICSD_9_5_153.png' 'GICSD_6_9_273'
img = cv2.imread(dataset_path+img_name)
# 1st approach, median in color
img_medianclr = cv2.medianBlur(img, 17)
img_medianclr_gray = cv2.cvtColor(img_medianclr, cv2.COLOR_RGB2GRAY)
# 2st approach, median in gray
img_gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
img_gray_median = cv2.medianBlur(img_gray, 17)
plt.subplot(231),plt.imshow(img)
plt.title('Original Image')
plt.subplot(232),plt.imshow(img_medianclr)
plt.title('Colored Median')
plt.subplot(233),plt.imshow(img_medianclr_gray, cmap='gray')
plt.title('Gray')

plt.subplot(234),plt.imshow(img)
plt.title('Original Image')
plt.subplot(235),plt.imshow(img_gray, cmap='gray')
plt.title('Gray')
plt.subplot(236),plt.imshow(img_gray_median, cmap='gray')
plt.title('Gray Median')

plt.show()<jupyter_output><empty_output><jupyter_text>* Our original assumption on denoising in the color first and then convert to single channel grayscale seems to hold<jupyter_code>plt.subplot(121),plt.imshow(img_medianclr_gray, cmap='gray')
plt.title('Color Median converted to gray')
plt.show()
plt.subplot(121),plt.imshow(img_gray_median, cmap='gray')
plt.title('Gray Median')
plt.show()<jupyter_output><empty_output><jupyter_text>Conclusions Denoising Salt and Pepper noise from imagesWe investigated three different approaches to remove salt and pepper noise in color images. * The Best results are acquired when using NLmean assuming seqeunce of images.    * This is the slowest while it may break case study assumptions* Second best is when using NLmean on one frame    * This is second slowest while substantial scene information is removed along with the noise. For this case we believe the gain is not much greater to opt for this one rather than median filtering* Third is when using median in colored image.     * As the difference is not much greater for the computational cost we opt for this one* The assumption to first denoise the color image and then convert to grayscale to have the minimum loss of scene information seems to hold *  Also as it is not clear if during runtime, the input images for classification will be clear and without salt and pepper noise, we may change the median blur such that the result image to be similar with the ones used in training, thus helping the classifier  ID detection using traditional approachesWe notice that we could have some basic shape information in the pre-processed steps on our data investigation. Perhaps it would be miningful to go on traditional old faithful geometric computer vision approaches. There is a moderate concern that this will fail the images are too noisy and it is an indication that we will heavily need to engineer the problem. However before bringing out the big-guns of deep learning, we will take a step back and see if some simple edge detection and contour extraction could give something descent. * We will use existing opencv functionalities to extract contours* We will need first to binarize the grayscale image using standard techniques    * There is heavy noisy salt and pepper noise so this could mean that we need a dynamic way to change edge detection parameters* On the extract contours we will apply simple shape detection rules already applied found in opencv.     * we will use simple geometric rules, to extract shapes    * We will assume that the biggest contour belongs is the most likely object to be an ID    * We assume that a rectangle shape is a FULLY_VISIBLE ID    * A square or triangle a partially visible id    * Everything else is assumed to be empty scene<jupyter_code>import csv
class ShapeDetectorOCV:
    def __init__(self):
        pass

    def detect(self, c):
        peri = cv2.arcLength(c, True)
        approx = cv2.approxPolyDP(c, 0.04 * peri, True)
        shape = "unidentified"
        if len(approx) == 3:
            shape = "triangle"
        elif len(approx) == 4:
            (x, y, w, h) = cv2.boundingRect(approx)
            ar = w / float(h)
            if (ar >= 0.95 and ar <= 1.05):
                shape = "square"
            else:
                shape = "rectangle"
        return shape, peri, approx, c

def shape_to_class(argument): 
    switcher = { 
        'unidentified': 'NO_VISIBILITY', 
        'triangle': 'PARTIAL_VISIBILITY', 
        'square': 'PARTIAL_VISIBILITY', 
        'rectangle': 'FULL_VISIBILITY' 
    } 
    return switcher.get(argument, "nothing")<jupyter_output><empty_output><jupyter_text>* Get all the image paths<jupyter_code>base_image_dir = '../data/'
csv_dir = base_image_dir + 'gicsd_labels.csv'
images_paths = []
with open(csv_dir, newline='') as csvfile:
    labelreader = csv.reader(csvfile, delimiter=',', quotechar='"')
    next(labelreader)
    for row in labelreader:
        src = base_image_dir + 'images/' + row[0]
        images_paths.append(src)<jupyter_output><empty_output><jupyter_text>* Pick an idx out of those images<jupyter_code>idx_NO_VISIBILITY = 11
idx_FULL_VISIBILITY = 7
idx_PARTIAL_VISIBILITY = 2
idx_challenged_image=idx_FULL_VISIBILITY
print(f'ID detection on: {images_paths[idx_challenged_image]}')
img = cv2.imread(images_paths[idx_challenged_image])
# 1st approach, median in color
img_medianclr = cv2.medianBlur(img, 17)
img_medianclr_gray = cv2.cvtColor(img_medianclr, cv2.COLOR_RGB2GRAY)
_, binmask = cv2.threshold(img_medianclr_gray, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
_, contours, _ = cv2.findContours(binmask, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)
sorted_cnts = sorted(contours, key=lambda x: len(x), reverse=True)
plt.subplot(121),plt.imshow(img_medianclr_gray, cmap='gray')
plt.title('Color Median converted to gray')
plt.subplot(122),plt.imshow(binmask, cmap='gray')
plt.title('binmask')
plt.show()<jupyter_output><empty_output><jupyter_text>* Detect Shape on using largest contour<jupyter_code>sd = ShapeDetectorOCV()
shape, peri, approx, c = 'unidentified', [], [], []
cX, cY = 10,20#img.shape[0]//2, img.shape[1]//2
if len(sorted_cnts) > 0:
    shape, _, approx, c = sd.detect(sorted_cnts[0])
cls = shape_to_class(shape)
print(cls)
img_res=cv2.drawContours(img.copy(), [c], 0, (255, 255, 0), 2)
img_res=cv2.drawContours(img_res, [approx], 0, (0, 255, 0), 2)
img_res=cv2.putText(img_res, cls, (cX, cY), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 255, 0), 2)
plt.subplot(121),plt.imshow(img_res, cmap='gray')
plt.title('Result')<jupyter_output><empty_output><jupyter_text>Short discussion on traditional approaches and next steps to deep learning* Using traditional approaches we will require a lot of assumptions, heuristics and geomatrically engineering the problem. * We got some initial results with some success but it looks that this path we will be better of trying out machine learning. * Given the nature of the     * preprocessed images (Similar in simplicity to MNIST digits),     * the resources (my personal laptop)     * and the time constraints (estimated a 10 hour study),     * we decide to use a simple neural network architecutre out of the box* Resnet18 looks to be a sensible first choice as it is fast to train even on CPU.     * We could consider transfer learning on a pretrained MNIST dataset as well rather than training from scratch    * Digits have rather simple shapes, while IDs are basically rectangles.    * MNIST dataset is basically simpled shapes/contours objects* EfficientNet (picking between B0-B7 based on target hw and accuracy) would be something I would go for to experiment for a production* Unfortunatelly the default implenentations (to my knowledge) come with 3 input channel requirement. So do the majority of the pre-trained networks* This leads create something from scratch - I will go to a simple vgg like implementation due to time constrains* I would have gone with FastAI but this time I will go with a Tensorlfow/Keras recipy    * I do have some notebooks for MNIST in FastAI however under the hood they load the original grayscale Image as 3channel so it is not good for this excersize  New Data Set generation for debug/fallback purposes* Under the afore findings, we cane enerate a new dataset of grayscale images using the denoising techniques discussed - for debug or fall back purposes* We Split the data into the 3 folders, one for each class* We make sure that there is no data loss during saving and loading using the necessary opencv flags, while we load a random grayscale image for sanity check* Please note that:    * this dataset is for visualization/debug puproses of what would be the training data - some additional steps (normalizations, dataaugmentations, reshape(width,height,1)) are done before training    * a fallback dataset, if due to time contrains, a pipeline for a "just in time" generation of training dataset  from original csv annotation file  and images is not physible<jupyter_code>import os
import csv
from shutil import copyfile

base_image_dir = '../data/'
csv_dir = base_image_dir+'gicsd_labels.csv'
out_image_dir = '../data/exploration/'

with open(csv_dir, newline='') as csvfile:
    labelreader = csv.reader(csvfile, delimiter=',', quotechar='"')
    next(labelreader)
    for row in labelreader:
        src = base_image_dir+'images/'+row[0]
        dst = out_image_dir+row[1].strip()+"/"+row[0]
        print(f'from {src} to {dst}')
        os.makedirs(os.path.dirname(dst), exist_ok=True)
        img = cv2.imread(src)
        # 1st approach, median in color
        img_medianclr = cv2.medianBlur(img, 17)
        img_medianclr_gray = cv2.cvtColor(img_medianclr, cv2.COLOR_RGB2GRAY)
        cv2.imwrite(dst, img_medianclr_gray, [cv2.IMWRITE_PNG_COMPRESSION, 0])  
        #copyfile(src, dst)<jupyter_output>from ../data/images/GICSD_1_0_3.png to ../data/exploration/FULL_VISIBILITY/GICSD_1_0_3.png
from ../data/images/GICSD_1_0_13.png to ../data/exploration/FULL_VISIBILITY/GICSD_1_0_13.png
from ../data/images/GICSD_1_1_33.png to ../data/exploration/PARTIAL_VISIBILITY/GICSD_1_1_33.png
from ../data/images/GICSD_1_1_53.png to ../data/exploration/FULL_VISIBILITY/GICSD_1_1_53.png
from ../data/images/GICSD_1_2_73.png to ../data/exploration/FULL_VISIBILITY/GICSD_1_2_73.png
from ../data/images/GICSD_1_3_93.png to ../data/exploration/FULL_VISIBILITY/GICSD_1_3_93.png
from ../data/images/GICSD_1_3_113.png to ../data/exploration/FULL_VISIBILITY/GICSD_1_3_113.png
from ../data/images/GICSD_1_4_133.png to ../data/exploration/FULL_VISIBILITY/GICSD_1_4_133.png
from ../data/images/GICSD_1_5_153.png to ../data/exploration/FULL_VISIBILITY/GICSD_1_5_153.png
from ../data/images/GICSD_1_5_173.png to ../data/exploration/FULL_VISIBILITY/GICSD_1_5_173.png
from ../data/images/GICSD_1_6_193.png to ../data/exploration/[...]<jupyter_text>* Some sanity test to with regards to saving/loading grayscale images using opencv<jupyter_code>img_path='../data/exploration/FULL_VISIBILITY/GICSD_50_9_293.png'
test_img=cv2.imread(img_path)
test_img_gray=cv2.imread(img_path, cv2.IMREAD_UNCHANGED)
print(test_img.shape)
print(test_img_gray.shape)<jupyter_output>(192, 192, 3)
(192, 192)
{'fraction_non_alphanumeric': 0.0775912551239508, 'fraction_numerical': 0.020593402303337886, 'mean_word_length': 5.057641146911026, 'pattern_counts': {'":': 0, '<': 66, '<?xml version=': 0, '>': 67, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '624963', 'license': 'MIT', 'n_tokens_mistral': 6621, 'n_tokens_neox': 6094, 'n_words': 2374, 'path': 'notebooks/a_data_exploration.ipynb', 'repo_name': 'antelidakis/dlcv_id_challenge', 'size': '20492'}